Max level shown:
12025-04-30T06:25:19.951ZDEBGtest_replacement_sanity (init_with_steps): Running step start_crdb
22025-04-30T06:25:19.952ZDEBGtest_replacement_sanity: Starting CRDB
32025-04-30T06:25:19.984ZINFOtest_replacement_sanity: cockroach temporary directory: /var/tmp/omicron_tmp/.tmp0v4YD4
42025-04-30T06:25:19.984ZINFOtest_replacement_sanity: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmp0v4YD4/data)
52025-04-30T06:25:20.004ZINFOtest_replacement_sanity: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmp0v4YD4/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmp0v4YD4/listen-url
62025-04-30T06:25:20.019ZINFOtest_replacement_sanity: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=6 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:25:21.507ZINFOtest_replacement_sanity: cockroach pid: 21711
82025-04-30T06:25:21.507ZINFOtest_replacement_sanity: cockroach listen URL: postgresql://root@[::1]:46323/omicron?sslmode=disable
92025-04-30T06:25:21.508ZDEBGtest_replacement_sanity (init_with_steps): Running step start_clickhouse
102025-04-30T06:25:21.508ZDEBGtest_replacement_sanity: Starting Clickhouse
112025-04-30T06:25:22.058ZDEBGtest_replacement_sanity (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:25:22.058ZDEBGtest_replacement_sanity: Starting Management Gateway
132025-04-30T06:25:22.187ZDEBGtest_replacement_sanity (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:25:22.187ZDEBGtest_replacement_sanity: Starting Dendrite for switch0
152025-04-30T06:25:22.259ZDEBGtest_replacement_sanity (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:25:22.259ZDEBGtest_replacement_sanity: Starting mgd for switch0
172025-04-30T06:25:22.374ZDEBGtest_replacement_sanity: mgd port is 35157
182025-04-30T06:25:22.374ZDEBGtest_replacement_sanity (init_with_steps): Running step record_switch_dns
192025-04-30T06:25:22.374ZDEBGtest_replacement_sanity: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:25:22.374ZDEBGtest_replacement_sanity (init_with_steps): Running step start_gateway_switch1
212025-04-30T06:25:22.374ZDEBGtest_replacement_sanity: Starting Management Gateway
222025-04-30T06:25:22.493ZDEBGtest_replacement_sanity (init_with_steps): Running step start_dendrite_switch1
232025-04-30T06:25:22.493ZDEBGtest_replacement_sanity: Starting Dendrite for switch1
242025-04-30T06:25:22.558ZDEBGtest_replacement_sanity (init_with_steps): Running step start_mgd_switch1
252025-04-30T06:25:22.558ZDEBGtest_replacement_sanity: Starting mgd for switch1
262025-04-30T06:25:22.678ZDEBGtest_replacement_sanity: mgd port is 44212
272025-04-30T06:25:22.678ZDEBGtest_replacement_sanity (init_with_steps): Running step record_switch_dns
282025-04-30T06:25:22.678ZDEBGtest_replacement_sanity: Recording DNS for the switch zones sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 switch_location = switch1
292025-04-30T06:25:22.678ZDEBGtest_replacement_sanity (init_with_steps): Running step start_internal_dns
302025-04-30T06:25:22.678ZINFOtest_replacement_sanity (store): opening sled database path = /var/tmp/omicron_tmp/.tmpDBpfLD
312025-04-30T06:25:22.786ZINFOtest_replacement_sanity (store): pruning trees for generations newer than 0
322025-04-30T06:25:22.786ZINFOtest_replacement_sanity (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:25:22.786ZINFOtest_replacement_sanity (dns): DNS server bound to address kind = dns local_address = [::1]:35287
342025-04-30T06:25:22.787ZDEBGtest_replacement_sanity (http): registered endpoint kind = dns local_addr = [::1]:34917 method = GET path = /config versions = all
352025-04-30T06:25:22.787ZDEBGtest_replacement_sanity (http): registered endpoint kind = dns local_addr = [::1]:34917 method = PUT path = /config versions = all
362025-04-30T06:25:22.787ZINFOtest_replacement_sanity (http): listening kind = dns local_addr = [::1]:34917
372025-04-30T06:25:22.789ZDEBGtest_replacement_sanity (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:34917
382025-04-30T06:25:22.789ZDEBGtest_replacement_sanity (init_with_steps): Running step start_external_dns
392025-04-30T06:25:22.789ZINFOtest_replacement_sanity (store): opening sled database path = /var/tmp/omicron_tmp/.tmpSUdChf
402025-04-30T06:25:22.801ZINFOtest_replacement_sanity (store): pruning trees for generations newer than 0
412025-04-30T06:25:22.801ZINFOtest_replacement_sanity (store): pruning trees for generations older than 0 keep = 3
422025-04-30T06:25:22.801ZINFOtest_replacement_sanity (dns): DNS server bound to address kind = dns local_address = [::1]:34273
432025-04-30T06:25:22.802ZDEBGtest_replacement_sanity (http): registered endpoint kind = dns local_addr = [::1]:35427 method = GET path = /config versions = all
442025-04-30T06:25:22.802ZDEBGtest_replacement_sanity (http): registered endpoint kind = dns local_addr = [::1]:35427 method = PUT path = /config versions = all
452025-04-30T06:25:22.802ZINFOtest_replacement_sanity (http): listening kind = dns local_addr = [::1]:35427
462025-04-30T06:25:22.803ZDEBGtest_replacement_sanity (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:35427
472025-04-30T06:25:22.803ZDEBGtest_replacement_sanity (init_with_steps): Running step start_nexus_internal
482025-04-30T06:25:22.803ZDEBGtest_replacement_sanity: Starting Nexus (internal API)
492025-04-30T06:25:22.804ZINFOtest_replacement_sanity: setting up nexus server
502025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Action
512025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = AnyActor
522025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = AuthenticatedActor
532025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = BlueprintConfig
542025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Database
552025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = DnsConfig
562025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Fleet
572025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Inventory
582025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = IpPoolList
592025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = ConsoleSessionList
602025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = DeviceAuthRequestList
612025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SiloCertificateList
622025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SiloIdentityProviderList
632025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SiloUserList
642025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = TargetReleaseConfig
652025-04-30T06:25:22.806ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = WebhookEventClassList
662025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Project
672025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Disk
682025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Snapshot
692025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = ProjectImage
702025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = AffinityGroup
712025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = AntiAffinityGroup
722025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Instance
732025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = IpPool
742025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = InstanceNetworkInterface
752025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Vpc
762025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = VpcRouter
772025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = InternetGateway
782025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = InternetGatewayIpPool
792025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = InternetGatewayIpAddress
802025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = RouterRoute
812025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = VpcSubnet
822025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = FloatingIp
832025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Image
842025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SiloImage
852025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = AddressLot
862025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Blueprint
872025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = LoopbackAddress
882025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Certificate
892025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = ConsoleSession
902025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = DeviceAuthRequest
912025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = DeviceAccessToken
922025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = PhysicalDisk
932025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Rack
942025-04-30T06:25:22.807ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = RoleBuiltin
952025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SshKey
962025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Silo
972025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SiloUser
982025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SiloGroup
992025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SupportBundle
1002025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = IdentityProvider
1012025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = SamlIdentityProvider
1022025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Sled
1032025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = TufRepo
1042025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = TufArtifact
1052025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = WebhookEvent
1062025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = WebhookReceiver
1072025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = WebhookSecret
1082025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Zpool
1092025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = Service
1102025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): registering Oso class class = UserBuiltin
1112025-04-30T06:25:22.808ZINFOtest_replacement_sanity (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1122025-04-30T06:25:22.830ZINFOtest_replacement_sanity (ServerContext): Setting up resolver using DNS address: [::1]:35287
1132025-04-30T06:25:22.830ZINFOtest_replacement_sanity (DnsResolver): new DNS resolver addresses = [[::1]:35287]
1142025-04-30T06:25:22.830ZINFOtest_replacement_sanity (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:46323/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 46323,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1152025-04-30T06:25:22.832ZDEBGtest_replacement_sanity (ServerContext): registered USDT probes
1162025-04-30T06:25:22.867ZINFOtest_replacement_sanity (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1172025-04-30T06:25:22.950ZDEBGtest_replacement_sanity (clickhouse-client): registered USDT probes id = 5b367921-bef4-45c5-976e-b05407d1b64c
1182025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /bgtasks versions = all
1192025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /bgtasks/activate versions = all
1202025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1212025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /clickhouse/policy versions = all
1222025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /clickhouse/policy versions = all
1232025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1242025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1252025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1262025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1272025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1282025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /demo-saga versions = all
1292025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1302025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /deployment/blueprints/all versions = all
1312025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1322025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1332025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /deployment/blueprints/import versions = all
1342025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /deployment/blueprints/regenerate versions = all
1352025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /deployment/blueprints/target versions = all
1362025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /deployment/blueprints/target versions = all
1372025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = PUT path = /deployment/blueprints/target/enabled versions = all
1382025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1392025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = PUT path = /disks/{disk_id} versions = all
1402025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /experimental/v1/system/support-bundles versions = all
1412025-04-30T06:25:23.084ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /experimental/v1/system/support-bundles versions = all
1422025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1432025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1442025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1452025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1462025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1472025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1482025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1492025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /instances/{instance_id}/migrate versions = all
1502025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /metrics/collectors versions = all
1512025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1522025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /metrics/producers versions = all
1532025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /mgs-updates versions = all
1542025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1552025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /oximeter/read-policy versions = all
1562025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /oximeter/read-policy versions = all
1572025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /physical-disk/expunge versions = all
1582025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /probes/{sled} versions = all
1592025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1602025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /sagas versions = all
1612025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /sagas/{saga_id} versions = all
1622025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /sled-agents/{sled_id} versions = all
1632025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /sled-agents/{sled_id} versions = all
1642025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1652025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /sleds/add versions = all
1662025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /sleds/expunge versions = all
1672025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /sleds/uninitialized versions = all
1682025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = PUT path = /switch/{switch_id} versions = all
1692025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = GET path = /v1/ping versions = all
1702025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = PUT path = /vmms/{propolis_id} versions = all
1712025-04-30T06:25:23.085ZDEBGtest_replacement_sanity (dropshot_internal): registered endpoint local_addr = 127.0.0.1:43527 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1722025-04-30T06:25:23.085ZINFOtest_replacement_sanity (dropshot_internal): listening local_addr = 127.0.0.1:43527
1732025-04-30T06:25:23.086ZDEBGtest_replacement_sanity (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:43527
1742025-04-30T06:25:23.086ZINFOtest_replacement_sanity (SEC): SEC running sec_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
1752025-04-30T06:25:23.088ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1762025-04-30T06:25:23.088ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
1772025-04-30T06:25:23.091ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1782025-04-30T06:25:23.092ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in users
1792025-04-30T06:25:23.094ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1802025-04-30T06:25:23.094ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
1812025-04-30T06:25:23.094ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1822025-04-30T06:25:23.094ZINFOtest_replacement_sanity (MgsUpdateDriver): starting MgsUpdateDriver
1832025-04-30T06:25:23.110ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: f6 ef 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 ............._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:49591 req_id = 1ad12468-a6b9-4024-9b7d-981d202d05ac
1842025-04-30T06:25:23.110ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63215,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49591 req_id = 1ad12468-a6b9-4024-9b7d-981d202d05ac
1852025-04-30T06:25:23.112ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63215, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49591 req_id = 1ad12468-a6b9-4024-9b7d-981d202d05ac
1862025-04-30T06:25:23.167ZINFOtest_replacement_sanity (DataLoader): created 5 built-in users
1872025-04-30T06:25:23.167ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1882025-04-30T06:25:23.167ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
1892025-04-30T06:25:23.169ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1902025-04-30T06:25:23.169ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in roles
1912025-04-30T06:25:23.169ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1922025-04-30T06:25:23.169ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
1932025-04-30T06:25:23.170ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1942025-04-30T06:25:23.194ZINFOtest_replacement_sanity (DataLoader): created 10 built-in roles
1952025-04-30T06:25:23.194ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1962025-04-30T06:25:23.195ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
1972025-04-30T06:25:23.201ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1982025-04-30T06:25:23.201ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in role assignments
1992025-04-30T06:25:23.201ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2002025-04-30T06:25:23.201ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2012025-04-30T06:25:23.202ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2022025-04-30T06:25:23.422ZINFOtest_replacement_sanity (DataLoader): created 4 built-in role assignments
2032025-04-30T06:25:23.422ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2042025-04-30T06:25:23.422ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2052025-04-30T06:25:23.424ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2062025-04-30T06:25:23.424ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in silos
2072025-04-30T06:25:23.424ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2082025-04-30T06:25:23.424ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2092025-04-30T06:25:23.424ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2102025-04-30T06:25:23.494ZINFOtest_replacement_sanity (DataLoader): created 2 built-in silos
2112025-04-30T06:25:23.494ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2122025-04-30T06:25:23.494ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2132025-04-30T06:25:23.495ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2142025-04-30T06:25:23.504ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:25:23.505ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:25:23.526ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:25:23.526ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in projects
2182025-04-30T06:25:23.526ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:25:23.526ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:25:23.527ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:25:23.538ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2222025-04-30T06:25:23.538ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2232025-04-30T06:25:23.538ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2242025-04-30T06:25:23.539ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2252025-04-30T06:25:23.539ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2262025-04-30T06:25:23.634ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2272025-04-30T06:25:23.634ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2282025-04-30T06:25:23.634ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2292025-04-30T06:25:23.635ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2302025-04-30T06:25:23.638ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:25:23.642ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2322025-04-30T06:25:23.642ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2332025-04-30T06:25:23.642ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2342025-04-30T06:25:23.642ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:25:23.642ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:25:23.642ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:25:23.644ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2382025-04-30T06:25:23.644ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:25:23.644ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:25:23.645ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:25:23.651ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2422025-04-30T06:25:23.655ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2432025-04-30T06:25:23.667ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2442025-04-30T06:25:23.667ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2452025-04-30T06:25:23.668ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2462025-04-30T06:25:23.724ZINFOtest_replacement_sanity (DataLoader): created built-in services project
2472025-04-30T06:25:23.725ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2482025-04-30T06:25:23.725ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2492025-04-30T06:25:23.728ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2502025-04-30T06:25:23.728ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in VPCs
2512025-04-30T06:25:23.728ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2522025-04-30T06:25:23.729ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2532025-04-30T06:25:23.729ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2542025-04-30T06:25:23.773ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:25:23.773ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:25:23.777ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:25:23.798ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2582025-04-30T06:25:23.798ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2592025-04-30T06:25:23.798ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2602025-04-30T06:25:23.798ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2612025-04-30T06:25:23.799ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2622025-04-30T06:25:23.801ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2632025-04-30T06:25:23.801ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2642025-04-30T06:25:23.801ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2652025-04-30T06:25:23.803ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2662025-04-30T06:25:23.807ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2672025-04-30T06:25:23.808ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2682025-04-30T06:25:23.809ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2692025-04-30T06:25:23.811ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2702025-04-30T06:25:23.820ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2712025-04-30T06:25:23.828ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2722025-04-30T06:25:23.828ZTRACtest_replacement_sanity (DataLoader): successful lookup of siloed resource "Project" using built-in user
2732025-04-30T06:25:23.828ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2742025-04-30T06:25:23.828ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2752025-04-30T06:25:23.828ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2762025-04-30T06:25:23.828ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2772025-04-30T06:25:23.828ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2782025-04-30T06:25:23.835ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2792025-04-30T06:25:23.835ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:25:23.835ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:25:23.841ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:25:23.852ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2832025-04-30T06:25:23.853ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2842025-04-30T06:25:23.853ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2852025-04-30T06:25:23.853ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2862025-04-30T06:25:23.868ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2872025-04-30T06:25:23.884ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2882025-04-30T06:25:23.884ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:25:23.884ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:25:23.885ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:25:23.913ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2922025-04-30T06:25:23.913ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2932025-04-30T06:25:23.913ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2942025-04-30T06:25:23.913ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2952025-04-30T06:25:23.914ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2962025-04-30T06:25:23.923ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2972025-04-30T06:25:23.923ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:25:23.923ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:25:23.923ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:25:23.932ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3012025-04-30T06:25:23.932ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3022025-04-30T06:25:23.933ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3032025-04-30T06:25:23.933ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3042025-04-30T06:25:23.936ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:25:23.941ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3062025-04-30T06:25:23.941ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3072025-04-30T06:25:23.941ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3082025-04-30T06:25:23.942ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3092025-04-30T06:25:23.951ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3102025-04-30T06:25:23.951ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3112025-04-30T06:25:23.951ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3122025-04-30T06:25:23.962ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:25:23.966ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:25:23.972ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:25:24.009ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3162025-04-30T06:25:24.010ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3172025-04-30T06:25:24.010ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3182025-04-30T06:25:24.010ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3192025-04-30T06:25:24.010ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3202025-04-30T06:25:24.018ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3212025-04-30T06:25:24.018ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:25:24.018ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:25:24.020ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:25:24.025ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3252025-04-30T06:25:24.025ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3262025-04-30T06:25:24.026ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3272025-04-30T06:25:24.026ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3282025-04-30T06:25:24.040ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3292025-04-30T06:25:24.047ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3302025-04-30T06:25:24.047ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:25:24.047ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:25:24.049ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:25:24.109ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3342025-04-30T06:25:24.126ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3352025-04-30T06:25:24.126ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3362025-04-30T06:25:24.126ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3372025-04-30T06:25:24.127ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3382025-04-30T06:25:24.132ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3392025-04-30T06:25:24.133ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:25:24.133ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:25:24.133ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:25:24.141ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3432025-04-30T06:25:24.141ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3442025-04-30T06:25:24.141ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3452025-04-30T06:25:24.141ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3462025-04-30T06:25:24.147ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3472025-04-30T06:25:24.154ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3482025-04-30T06:25:24.154ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:25:24.154ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:25:24.155ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:25:24.228ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3522025-04-30T06:25:24.228ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3532025-04-30T06:25:24.228ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3542025-04-30T06:25:24.229ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3552025-04-30T06:25:24.230ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3562025-04-30T06:25:24.233ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3572025-04-30T06:25:24.233ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:25:24.233ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:25:24.234ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:25:24.238ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3612025-04-30T06:25:24.238ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:25:24.238ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:25:24.239ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:25:24.242ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3652025-04-30T06:25:24.251ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3662025-04-30T06:25:24.251ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3672025-04-30T06:25:24.251ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3682025-04-30T06:25:24.251ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3692025-04-30T06:25:24.263ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in VPC firewall rules
3702025-04-30T06:25:24.263ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3712025-04-30T06:25:24.263ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3722025-04-30T06:25:24.263ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3732025-04-30T06:25:24.268ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3742025-04-30T06:25:24.268ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3752025-04-30T06:25:24.268ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3762025-04-30T06:25:24.271ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:25:24.271ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:25:24.271ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:25:24.275ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3802025-04-30T06:25:24.275ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3812025-04-30T06:25:24.275ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3822025-04-30T06:25:24.275ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3832025-04-30T06:25:24.275ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3842025-04-30T06:25:24.283ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3852025-04-30T06:25:24.283ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3862025-04-30T06:25:24.283ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3872025-04-30T06:25:24.284ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3882025-04-30T06:25:24.286ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3892025-04-30T06:25:24.286ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3902025-04-30T06:25:24.286ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3912025-04-30T06:25:24.287ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3922025-04-30T06:25:24.322ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3932025-04-30T06:25:24.327ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3942025-04-30T06:25:24.327ZTRACtest_replacement_sanity (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3952025-04-30T06:25:24.328ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3962025-04-30T06:25:24.328ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3972025-04-30T06:25:24.328ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3982025-04-30T06:25:24.328ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
3992025-04-30T06:25:24.328ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4002025-04-30T06:25:24.331ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4012025-04-30T06:25:24.331ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:25:24.331ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:25:24.331ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:25:24.334ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4052025-04-30T06:25:24.334ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4062025-04-30T06:25:24.334ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4072025-04-30T06:25:24.334ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4082025-04-30T06:25:24.337ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4092025-04-30T06:25:24.347ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4102025-04-30T06:25:24.347ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:25:24.347ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:25:24.348ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:25:24.358ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4142025-04-30T06:25:24.358ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4152025-04-30T06:25:24.358ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4162025-04-30T06:25:24.358ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4172025-04-30T06:25:24.359ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4182025-04-30T06:25:24.361ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4192025-04-30T06:25:24.361ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:25:24.361ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:25:24.361ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:25:24.364ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4232025-04-30T06:25:24.364ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4242025-04-30T06:25:24.364ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4252025-04-30T06:25:24.365ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4262025-04-30T06:25:24.367ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4272025-04-30T06:25:24.372ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4282025-04-30T06:25:24.372ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4292025-04-30T06:25:24.372ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4302025-04-30T06:25:24.372ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4312025-04-30T06:25:24.508ZINFOtest_replacement_sanity (DataLoader): created built-in VPC firewall rules
4322025-04-30T06:25:24.508ZDEBGtest_replacement_sanity (DataLoader): attempting to create built-in VPC Subnets
4332025-04-30T06:25:24.508ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4342025-04-30T06:25:24.508ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4352025-04-30T06:25:24.509ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4362025-04-30T06:25:24.517ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4372025-04-30T06:25:24.517ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4382025-04-30T06:25:24.517ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4392025-04-30T06:25:24.520ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:25:24.520ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:25:24.521ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:25:24.525ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4432025-04-30T06:25:24.525ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4442025-04-30T06:25:24.525ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4452025-04-30T06:25:24.525ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4462025-04-30T06:25:24.526ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4472025-04-30T06:25:24.527ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4482025-04-30T06:25:24.528ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4492025-04-30T06:25:24.528ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4502025-04-30T06:25:24.528ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4512025-04-30T06:25:24.530ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4522025-04-30T06:25:24.530ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4532025-04-30T06:25:24.530ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4542025-04-30T06:25:24.530ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4552025-04-30T06:25:24.535ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:25:24.571ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4572025-04-30T06:25:24.571ZTRACtest_replacement_sanity (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4582025-04-30T06:25:24.571ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:25:24.571ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:25:24.572ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:25:24.580ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4622025-04-30T06:25:24.580ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4632025-04-30T06:25:24.580ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4642025-04-30T06:25:24.580ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4652025-04-30T06:25:24.581ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4662025-04-30T06:25:24.603ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4672025-04-30T06:25:24.603ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4682025-04-30T06:25:24.604ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4692025-04-30T06:25:24.605ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4702025-04-30T06:25:24.609ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4712025-04-30T06:25:24.609ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4722025-04-30T06:25:24.609ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4732025-04-30T06:25:24.611ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4742025-04-30T06:25:24.614ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:25:24.650ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4762025-04-30T06:25:24.671ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4772025-04-30T06:25:24.672ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4782025-04-30T06:25:24.672ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4792025-04-30T06:25:24.673ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4802025-04-30T06:25:24.673ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4812025-04-30T06:25:24.676ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4822025-04-30T06:25:24.676ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:25:24.676ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:25:24.677ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:25:24.682ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4862025-04-30T06:25:24.682ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4872025-04-30T06:25:24.683ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4882025-04-30T06:25:24.683ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4892025-04-30T06:25:24.686ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4902025-04-30T06:25:24.698ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4912025-04-30T06:25:24.699ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4922025-04-30T06:25:24.699ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4932025-04-30T06:25:24.699ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4942025-04-30T06:25:24.758ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:25:24.758ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:25:24.759ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:25:24.769ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4982025-04-30T06:25:24.769ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4992025-04-30T06:25:24.769ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5002025-04-30T06:25:24.770ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5012025-04-30T06:25:24.770ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5022025-04-30T06:25:24.778ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5032025-04-30T06:25:24.778ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5042025-04-30T06:25:24.778ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5052025-04-30T06:25:24.779ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5062025-04-30T06:25:24.798ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5072025-04-30T06:25:24.798ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5082025-04-30T06:25:24.798ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5092025-04-30T06:25:24.799ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5102025-04-30T06:25:24.809ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:25:24.815ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5122025-04-30T06:25:24.844ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5132025-04-30T06:25:24.844ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5142025-04-30T06:25:24.844ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5152025-04-30T06:25:24.844ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5162025-04-30T06:25:24.845ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5172025-04-30T06:25:24.850ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5182025-04-30T06:25:24.850ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:25:24.850ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:25:24.853ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:25:24.855ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5222025-04-30T06:25:24.856ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5232025-04-30T06:25:24.856ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5242025-04-30T06:25:24.856ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5252025-04-30T06:25:24.866ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5262025-04-30T06:25:24.873ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5272025-04-30T06:25:24.873ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5282025-04-30T06:25:24.873ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5292025-04-30T06:25:24.874ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5302025-04-30T06:25:24.891ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:25:24.891ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:25:24.892ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:25:24.931ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5342025-04-30T06:25:24.931ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5352025-04-30T06:25:24.931ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5362025-04-30T06:25:24.931ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5372025-04-30T06:25:24.932ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5382025-04-30T06:25:24.933ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5392025-04-30T06:25:24.934ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5402025-04-30T06:25:24.934ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5412025-04-30T06:25:24.934ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5422025-04-30T06:25:24.981ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5432025-04-30T06:25:24.981ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5442025-04-30T06:25:24.981ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5452025-04-30T06:25:24.982ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5462025-04-30T06:25:24.985ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:25:24.996ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5482025-04-30T06:25:25.017ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5492025-04-30T06:25:25.017ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5502025-04-30T06:25:25.018ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5512025-04-30T06:25:25.019ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5522025-04-30T06:25:25.020ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5532025-04-30T06:25:25.023ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5542025-04-30T06:25:25.023ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:25:25.023ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:25:25.024ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:25:25.032ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5582025-04-30T06:25:25.033ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5592025-04-30T06:25:25.034ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5602025-04-30T06:25:25.035ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5612025-04-30T06:25:25.056ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5622025-04-30T06:25:25.067ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5632025-04-30T06:25:25.067ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5642025-04-30T06:25:25.067ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5652025-04-30T06:25:25.068ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5662025-04-30T06:25:25.076ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:25:25.077ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:25:25.078ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:25:25.083ZINFOtest_replacement_sanity (DataLoader): created built-in services vpc subnets
5702025-04-30T06:25:25.083ZINFOtest_replacement_sanity (DataLoader): created built-in services vpc
5712025-04-30T06:25:25.083ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:25:25.083ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:25:25.085ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:25:25.085ZDEBGtest_replacement_sanity (DataLoader): attempting to create silo users
5752025-04-30T06:25:25.085ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:25:25.085ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:25:25.086ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:25:25.094ZINFOtest_replacement_sanity (DataLoader): created 2 silo users
5792025-04-30T06:25:25.095ZDEBGtest_replacement_sanity (DataLoader): attempting to create silo user role assignments
5802025-04-30T06:25:25.095ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5812025-04-30T06:25:25.096ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5822025-04-30T06:25:25.097ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5832025-04-30T06:25:25.101ZINFOtest_replacement_sanity (DataLoader): created 2 silo user role assignments
5842025-04-30T06:25:25.101ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5852025-04-30T06:25:25.101ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5862025-04-30T06:25:25.102ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5872025-04-30T06:25:25.109ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5882025-04-30T06:25:25.109ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5892025-04-30T06:25:25.110ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5902025-04-30T06:25:25.121ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5912025-04-30T06:25:25.122ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5922025-04-30T06:25:25.122ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5932025-04-30T06:25:25.122ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5942025-04-30T06:25:25.123ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5952025-04-30T06:25:25.131ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5962025-04-30T06:25:25.135ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5972025-04-30T06:25:25.135ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5982025-04-30T06:25:25.135ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
5992025-04-30T06:25:25.136ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6002025-04-30T06:25:25.143ZTRACtest_replacement_sanity (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
6012025-04-30T06:25:25.143ZTRACtest_replacement_sanity (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6022025-04-30T06:25:25.143ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6032025-04-30T06:25:25.143ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
6042025-04-30T06:25:25.144ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6052025-04-30T06:25:25.155ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
6062025-04-30T06:25:25.157ZDEBGtest_replacement_sanity (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
6072025-04-30T06:25:25.157ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6082025-04-30T06:25:25.157ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
6092025-04-30T06:25:25.159ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6102025-04-30T06:25:25.229ZTRACtest_replacement_sanity (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6112025-04-30T06:25:25.229ZDEBGtest_replacement_sanity (DataLoader): roles roles = RoleSet { roles: {} }
6122025-04-30T06:25:25.230ZDEBGtest_replacement_sanity (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6132025-04-30T06:25:25.233ZINFOtest_replacement_sanity (nexus): populate complete
6142025-04-30T06:25:25.233ZINFOtest_replacement_sanity (nexus): activating background tasks
6152025-04-30T06:25:25.237ZDEBGtest_replacement_sanity (init_with_steps): Running step start_sled1
6162025-04-30T06:25:25.237ZINFOtest_replacement_sanity (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6172025-04-30T06:25:25.288ZINFOtest_replacement_sanity (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6182025-04-30T06:25:25.288ZINFOtest_replacement_sanity (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6192025-04-30T06:25:25.289ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6202025-04-30T06:25:25.289ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6212025-04-30T06:25:25.289ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6222025-04-30T06:25:25.289ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6232025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6242025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6252025-04-30T06:25:25.290ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6262025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6272025-04-30T06:25:25.290ZWARNtest_replacement_sanity (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6282025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 109.147\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6292025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6302025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6312025-04-30T06:25:25.290ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6322025-04-30T06:25:25.290ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6332025-04-30T06:25:25.291ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6342025-04-30T06:25:25.291ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6352025-04-30T06:25:25.291ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6362025-04-30T06:25:25.291ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6372025-04-30T06:25:25.291ZWARNtest_replacement_sanity (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6382025-04-30T06:25:25.291ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 48.957\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6392025-04-30T06:25:25.291ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6402025-04-30T06:25:25.291ZINFOtest_replacement_sanity (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:15:25.291665717 UTC
6412025-04-30T06:25:25.291ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6422025-04-30T06:25:25.292ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6432025-04-30T06:25:25.292ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6442025-04-30T06:25:25.292ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6452025-04-30T06:25:25.292ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6462025-04-30T06:25:25.292ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6472025-04-30T06:25:25.292ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6482025-04-30T06:25:25.293ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6492025-04-30T06:25:25.293ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6502025-04-30T06:25:25.293ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6512025-04-30T06:25:25.293ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6522025-04-30T06:25:25.293ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6532025-04-30T06:25:25.294ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6542025-04-30T06:25:25.294ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6552025-04-30T06:25:25.294ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6562025-04-30T06:25:25.294ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6572025-04-30T06:25:25.295ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6582025-04-30T06:25:25.295ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6592025-04-30T06:25:25.295ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6602025-04-30T06:25:25.295ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6612025-04-30T06:25:25.295ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6622025-04-30T06:25:25.295ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6632025-04-30T06:25:25.295ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6642025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6652025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6662025-04-30T06:25:25.296ZWARNtest_replacement_sanity (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6672025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 63.316\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6682025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6692025-04-30T06:25:25.296ZWARNtest_replacement_sanity (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6702025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 40.36\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6712025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6722025-04-30T06:25:25.296ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6732025-04-30T06:25:25.296ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6742025-04-30T06:25:25.296ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6752025-04-30T06:25:25.296ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6762025-04-30T06:25:25.297ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6772025-04-30T06:25:25.297ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6782025-04-30T06:25:25.297ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6792025-04-30T06:25:25.297ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6802025-04-30T06:25:25.297ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6812025-04-30T06:25:25.297ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6822025-04-30T06:25:25.297ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6832025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6842025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 65.36\u{b5}s iteration = 1 status = {"error":"task disabled"}
6852025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6862025-04-30T06:25:25.298ZWARNtest_replacement_sanity (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6872025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 32.284\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6882025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6892025-04-30T06:25:25.298ZINFOtest_replacement_sanity (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6902025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 85.36\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6912025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6922025-04-30T06:25:25.298ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6932025-04-30T06:25:25.298ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6942025-04-30T06:25:25.298ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6952025-04-30T06:25:25.298ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6962025-04-30T06:25:25.299ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6972025-04-30T06:25:25.300ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6982025-04-30T06:25:25.300ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6992025-04-30T06:25:25.300ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7002025-04-30T06:25:25.300ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
7012025-04-30T06:25:25.300ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
7022025-04-30T06:25:25.300ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
7032025-04-30T06:25:25.301ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
7042025-04-30T06:25:25.301ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
7052025-04-30T06:25:25.301ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
7062025-04-30T06:25:25.302ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
7072025-04-30T06:25:25.302ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
7082025-04-30T06:25:25.302ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7092025-04-30T06:25:25.302ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7102025-04-30T06:25:25.304ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7112025-04-30T06:25:25.304ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7122025-04-30T06:25:25.304ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7132025-04-30T06:25:25.304ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7142025-04-30T06:25:25.306ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7152025-04-30T06:25:25.306ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7162025-04-30T06:25:25.306ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7172025-04-30T06:25:25.306ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7182025-04-30T06:25:25.306ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7192025-04-30T06:25:25.306ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7202025-04-30T06:25:25.307ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7212025-04-30T06:25:25.307ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7222025-04-30T06:25:25.307ZINFOtest_replacement_sanity (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7232025-04-30T06:25:25.307ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = instance_updater elapsed = 309.447\u{b5}s iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7242025-04-30T06:25:25.307ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7252025-04-30T06:25:25.307ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7262025-04-30T06:25:25.308ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7272025-04-30T06:25:25.308ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7282025-04-30T06:25:25.308ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7292025-04-30T06:25:25.308ZDEBGtest_replacement_sanity (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7302025-04-30T06:25:25.309ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7312025-04-30T06:25:25.309ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7322025-04-30T06:25:25.309ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7332025-04-30T06:25:25.309ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7342025-04-30T06:25:25.309ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7352025-04-30T06:25:25.309ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7362025-04-30T06:25:25.309ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7372025-04-30T06:25:25.310ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7382025-04-30T06:25:25.310ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7392025-04-30T06:25:25.311ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7402025-04-30T06:25:25.311ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7412025-04-30T06:25:25.311ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7422025-04-30T06:25:25.311ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7432025-04-30T06:25:25.312ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7442025-04-30T06:25:25.312ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7452025-04-30T06:25:25.312ZDEBGtest_replacement_sanity (SagaRecoverer): listing candidate sagas for recovery
7462025-04-30T06:25:25.312ZTRACtest_replacement_sanity (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7472025-04-30T06:25:25.312ZDEBGtest_replacement_sanity (SagaRecoverer): roles roles = RoleSet { roles: {} }
7482025-04-30T06:25:25.313ZDEBGtest_replacement_sanity (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7492025-04-30T06:25:25.313ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7502025-04-30T06:25:25.313ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7512025-04-30T06:25:25.313ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7522025-04-30T06:25:25.314ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7532025-04-30T06:25:25.314ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7542025-04-30T06:25:25.314ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7552025-04-30T06:25:25.314ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7562025-04-30T06:25:25.314ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7572025-04-30T06:25:25.314ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7582025-04-30T06:25:25.315ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7592025-04-30T06:25:25.315ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7602025-04-30T06:25:25.316ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7612025-04-30T06:25:25.316ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7622025-04-30T06:25:25.316ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7632025-04-30T06:25:25.316ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7642025-04-30T06:25:25.316ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7652025-04-30T06:25:25.317ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7662025-04-30T06:25:25.317ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7672025-04-30T06:25:25.317ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7682025-04-30T06:25:25.317ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7692025-04-30T06:25:25.318ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7702025-04-30T06:25:25.318ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7712025-04-30T06:25:25.318ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7722025-04-30T06:25:25.318ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7732025-04-30T06:25:25.318ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7742025-04-30T06:25:25.318ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7752025-04-30T06:25:25.318ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7762025-04-30T06:25:25.318ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7772025-04-30T06:25:25.318ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7782025-04-30T06:25:25.319ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7792025-04-30T06:25:25.319ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7802025-04-30T06:25:25.319ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7812025-04-30T06:25:25.319ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7822025-04-30T06:25:25.320ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7832025-04-30T06:25:25.320ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7842025-04-30T06:25:25.320ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7852025-04-30T06:25:25.320ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7862025-04-30T06:25:25.321ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7872025-04-30T06:25:25.322ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpuK8mDj/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7882025-04-30T06:25:25.322ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpuK8mDj/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7892025-04-30T06:25:25.322ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpoFsHo0/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7902025-04-30T06:25:25.322ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpoFsHo0/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7912025-04-30T06:25:25.377ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:46651 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:25:25.377ZINFOtest_replacement_sanity (dropshot (Repo Depot)): listening local_addr = [::1]:46651 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7932025-04-30T06:25:25.378ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:46651 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7942025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:25:25.384ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8552025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8562025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8572025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8582025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8592025-04-30T06:25:25.385ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8602025-04-30T06:25:25.386ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8612025-04-30T06:25:25.386ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8622025-04-30T06:25:25.386ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35343 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8632025-04-30T06:25:25.386ZINFOtest_replacement_sanity (dropshot): listening local_addr = [::1]:35343 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8642025-04-30T06:25:25.386ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:35343 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8652025-04-30T06:25:25.386ZDEBGtest_replacement_sanity (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8662025-04-30T06:25:25.386ZDEBGtest_replacement_sanity (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:43527/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8672025-04-30T06:25:25.387ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 01 6b 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 .k..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:57186 req_id = 7c6159de-3117-4d03-bbbb-e89daa888f08
8682025-04-30T06:25:25.387ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 363,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57186 req_id = 7c6159de-3117-4d03-bbbb-e89daa888f08
8692025-04-30T06:25:25.387ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 363, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57186 req_id = 7c6159de-3117-4d03-bbbb-e89daa888f08
8702025-04-30T06:25:25.387ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 62 c0 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 b............_ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49182 req_id = 2957ccfd-dcf5-4038-b04c-6958752d641d
8712025-04-30T06:25:25.387ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25280,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49182 req_id = 2957ccfd-dcf5-4038-b04c-6958752d641d
8722025-04-30T06:25:25.387ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 25280, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49182 req_id = 2957ccfd-dcf5-4038-b04c-6958752d641d
8732025-04-30T06:25:25.388ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
8742025-04-30T06:25:25.388ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8752025-04-30T06:25:25.388ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
8762025-04-30T06:25:25.388ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
8772025-04-30T06:25:25.388ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
8782025-04-30T06:25:25.388ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:45181
8792025-04-30T06:25:25.389ZWARNtest_replacement_sanity (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8802025-04-30T06:25:25.389ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 99.120552ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8812025-04-30T06:25:25.389ZWARNtest_replacement_sanity (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8822025-04-30T06:25:25.389ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 98.024287ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8832025-04-30T06:25:25.390ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8842025-04-30T06:25:25.390ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8852025-04-30T06:25:25.390ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8862025-04-30T06:25:25.390ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8872025-04-30T06:25:25.391ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8882025-04-30T06:25:25.392ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:45181 req_id = d37b8543-4fba-4fde-94b0-32f285567f6d uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8892025-04-30T06:25:25.395ZINFOtest_replacement_sanity (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8902025-04-30T06:25:25.404ZINFOtest_replacement_sanity (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:15:25.291665717 UTC pruned = {}
8912025-04-30T06:25:25.405ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 113.236817ms iteration = 1 status = {"expiration":"2025-04-30T06:15:25.291665717Z","pruned":[]}
8922025-04-30T06:25:25.407ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8932025-04-30T06:25:25.409ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 57 5b 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 W[..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:55721 req_id = 13ba08a4-7179-421b-9b29-ab31528e988b
8942025-04-30T06:25:25.410ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22363,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55721 req_id = 13ba08a4-7179-421b-9b29-ab31528e988b
8952025-04-30T06:25:25.410ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22363, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55721 req_id = 13ba08a4-7179-421b-9b29-ab31528e988b
8962025-04-30T06:25:25.410ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8972025-04-30T06:25:25.421ZERROtest_replacement_sanity (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8982025-04-30T06:25:25.421ZERROtest_replacement_sanity (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8992025-04-30T06:25:25.421ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 128.599176ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9002025-04-30T06:25:25.423ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: d5 ce 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62200 req_id = 02b5c570-9164-4244-8118-78f50ebec581
9012025-04-30T06:25:25.423ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54734,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62200 req_id = 02b5c570-9164-4244-8118-78f50ebec581
9022025-04-30T06:25:25.426ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54734, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62200 req_id = 02b5c570-9164-4244-8118-78f50ebec581
9032025-04-30T06:25:25.426ZERROtest_replacement_sanity (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9042025-04-30T06:25:25.426ZERROtest_replacement_sanity (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9052025-04-30T06:25:25.426ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 131.001662ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9062025-04-30T06:25:25.426ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9072025-04-30T06:25:25.434ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9082025-04-30T06:25:25.434ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9092025-04-30T06:25:25.434ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9102025-04-30T06:25:25.434ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9112025-04-30T06:25:25.436ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 141.088229ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
9122025-04-30T06:25:25.473ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9132025-04-30T06:25:25.477ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9142025-04-30T06:25:25.477ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9152025-04-30T06:25:25.477ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9162025-04-30T06:25:25.478ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9172025-04-30T06:25:25.509ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9182025-04-30T06:25:25.527ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9192025-04-30T06:25:25.527ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9202025-04-30T06:25:25.527ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9212025-04-30T06:25:25.528ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9222025-04-30T06:25:25.529ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9232025-04-30T06:25:25.538ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
9242025-04-30T06:25:25.538ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
9252025-04-30T06:25:25.538ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
9262025-04-30T06:25:25.538ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
9272025-04-30T06:25:25.539ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9282025-04-30T06:25:25.545ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9292025-04-30T06:25:25.546ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9302025-04-30T06:25:25.546ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9312025-04-30T06:25:25.546ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9322025-04-30T06:25:25.582ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9332025-04-30T06:25:25.582ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9342025-04-30T06:25:25.583ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9352025-04-30T06:25:25.606ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9362025-04-30T06:25:25.610ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9372025-04-30T06:25:25.610ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9382025-04-30T06:25:25.610ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9392025-04-30T06:25:25.611ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9402025-04-30T06:25:25.611ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9412025-04-30T06:25:25.644ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9422025-04-30T06:25:25.669ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9432025-04-30T06:25:25.672ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9442025-04-30T06:25:25.673ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9452025-04-30T06:25:25.673ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9462025-04-30T06:25:25.704ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9472025-04-30T06:25:25.704ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9482025-04-30T06:25:25.704ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9492025-04-30T06:25:25.705ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9502025-04-30T06:25:25.706ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9512025-04-30T06:25:25.706ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9522025-04-30T06:25:25.707ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9532025-04-30T06:25:25.707ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9542025-04-30T06:25:25.714ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9552025-04-30T06:25:25.714ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9562025-04-30T06:25:25.714ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9572025-04-30T06:25:25.715ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9582025-04-30T06:25:25.850ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 536.680841ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9592025-04-30T06:25:25.850ZTRACtest_replacement_sanity (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9602025-04-30T06:25:25.850ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9612025-04-30T06:25:25.850ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9622025-04-30T06:25:25.852ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9632025-04-30T06:25:25.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9642025-04-30T06:25:25.901ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9652025-04-30T06:25:25.901ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9662025-04-30T06:25:26.011ZDEBGtest_replacement_sanity (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9672025-04-30T06:25:26.011ZINFOtest_replacement_sanity (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9682025-04-30T06:25:26.011ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 700.144412ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9692025-04-30T06:25:26.011ZINFOtest_replacement_sanity (SagaRecoverer): listed in-progress sagas count = 0
9702025-04-30T06:25:26.011ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 699.267575ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9712025-04-30T06:25:26.016ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9722025-04-30T06:25:26.022ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9732025-04-30T06:25:26.022ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9742025-04-30T06:25:26.022ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9752025-04-30T06:25:26.022ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9762025-04-30T06:25:26.022ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9772025-04-30T06:25:26.022ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9782025-04-30T06:25:26.035ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 716.662232ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9792025-04-30T06:25:26.035ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9802025-04-30T06:25:26.035ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9812025-04-30T06:25:26.036ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9822025-04-30T06:25:26.062ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 747.866375ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9832025-04-30T06:25:26.080ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 763.591416ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9842025-04-30T06:25:26.099ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 779.158593ms iteration = 1 status = {"by_rx":{},"error":null}
9852025-04-30T06:25:26.099ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9862025-04-30T06:25:26.099ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9872025-04-30T06:25:26.100ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9882025-04-30T06:25:26.101ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9892025-04-30T06:25:26.112ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9902025-04-30T06:25:26.118ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9912025-04-30T06:25:26.126ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
9922025-04-30T06:25:26.209ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9932025-04-30T06:25:26.210ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9942025-04-30T06:25:26.214ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9952025-04-30T06:25:26.214ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9962025-04-30T06:25:26.214ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9972025-04-30T06:25:26.214ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9982025-04-30T06:25:26.215ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9992025-04-30T06:25:26.220ZWARNtest_replacement_sanity (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
10002025-04-30T06:25:26.220ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 931.150626ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
10012025-04-30T06:25:26.220ZWARNtest_replacement_sanity (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
10022025-04-30T06:25:26.220ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 924.945683ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
10032025-04-30T06:25:26.221ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
10042025-04-30T06:25:26.221ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 922.518417ms iteration = 1 status = {"error":"inventory collection is None"}
10052025-04-30T06:25:26.222ZWARNtest_replacement_sanity (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
10062025-04-30T06:25:26.222ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 931.523459ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
10072025-04-30T06:25:26.222ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a3 95 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54544 req_id = 40cb981e-be3c-4525-a6a1-7922fa7507a8
10082025-04-30T06:25:26.222ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41877,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54544 req_id = 40cb981e-be3c-4525-a6a1-7922fa7507a8
10092025-04-30T06:25:26.223ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 41877, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54544 req_id = 40cb981e-be3c-4525-a6a1-7922fa7507a8
10102025-04-30T06:25:26.225ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
10112025-04-30T06:25:26.225ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 929.367364ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
10122025-04-30T06:25:26.232ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10132025-04-30T06:25:26.241ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10142025-04-30T06:25:26.241ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10152025-04-30T06:25:26.241ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10162025-04-30T06:25:26.242ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10172025-04-30T06:25:26.242ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10182025-04-30T06:25:26.242ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10192025-04-30T06:25:26.243ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10202025-04-30T06:25:26.264ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 964.193432ms iteration = 1 status = {}
10212025-04-30T06:25:26.269ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10222025-04-30T06:25:26.272ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10232025-04-30T06:25:26.272ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10242025-04-30T06:25:26.272ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10252025-04-30T06:25:26.277ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10262025-04-30T06:25:26.279ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10272025-04-30T06:25:26.279ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10282025-04-30T06:25:26.279ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10292025-04-30T06:25:26.279ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10302025-04-30T06:25:26.279ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10312025-04-30T06:25:26.306ZTRACtest_replacement_sanity (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10322025-04-30T06:25:26.306ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10332025-04-30T06:25:26.306ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10342025-04-30T06:25:26.307ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10352025-04-30T06:25:26.359ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10362025-04-30T06:25:26.359ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10372025-04-30T06:25:26.359ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10382025-04-30T06:25:26.359ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10392025-04-30T06:25:26.360ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10402025-04-30T06:25:26.394ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = region_replacement elapsed = 1.092004477s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10412025-04-30T06:25:26.395ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10422025-04-30T06:25:26.395ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10432025-04-30T06:25:26.396ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10442025-04-30T06:25:26.400ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.086516923s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10452025-04-30T06:25:26.404ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10462025-04-30T06:25:26.408ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
10472025-04-30T06:25:26.408ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10482025-04-30T06:25:26.408ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10492025-04-30T06:25:26.409ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10502025-04-30T06:25:26.455ZTRACtest_replacement_sanity (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
10512025-04-30T06:25:26.455ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 1.135875558s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
10522025-04-30T06:25:26.563ZINFOtest_replacement_sanity (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10532025-04-30T06:25:26.563ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.257111943s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10542025-04-30T06:25:26.575ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 1.259322789s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
10552025-04-30T06:25:26.594ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.290636458s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10562025-04-30T06:25:26.833ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10572025-04-30T06:25:26.843ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10582025-04-30T06:25:26.843ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10592025-04-30T06:25:26.843ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10602025-04-30T06:25:26.844ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10612025-04-30T06:25:26.845ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10622025-04-30T06:25:26.845ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10632025-04-30T06:25:26.845ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10642025-04-30T06:25:26.845ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10652025-04-30T06:25:26.845ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10662025-04-30T06:25:26.845ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10672025-04-30T06:25:26.845ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 1453131 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:45181 req_id = d37b8543-4fba-4fde-94b0-32f285567f6d response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10682025-04-30T06:25:26.846ZDEBGtest_replacement_sanity (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:43527/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "d37b8543-4fba-4fde-94b0-32f285567f6d", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10692025-04-30T06:25:26.846ZDEBGtest_replacement_sanity (init_with_steps): Running step start_sled2
10702025-04-30T06:25:26.847ZINFOtest_replacement_sanity (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
10712025-04-30T06:25:27.039ZINFOtest_replacement_sanity (SledAgent): created simulated sled agent server = 039be560-54cc-49e3-88df-1a29dadbf913 sim_mode = Explicit sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
10722025-04-30T06:25:27.039ZINFOtest_replacement_sanity (omicron_sled_agent::sim::SimulatedUpstairs): registered sled 039be560-54cc-49e3-88df-1a29dadbf913 storage
10732025-04-30T06:25:27.045ZINFOtest_replacement_sanity (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
10742025-04-30T06:25:27.045ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 1.748015917s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
10752025-04-30T06:25:27.118ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
10762025-04-30T06:25:27.118ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10772025-04-30T06:25:27.118ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10782025-04-30T06:25:27.118ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10792025-04-30T06:25:27.119ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10802025-04-30T06:25:27.122ZWARNtest_replacement_sanity (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
10812025-04-30T06:25:27.122ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 1.830339371s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
10822025-04-30T06:25:27.123ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
10832025-04-30T06:25:27.123ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10842025-04-30T06:25:27.123ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10852025-04-30T06:25:27.130ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10862025-04-30T06:25:27.188ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
10872025-04-30T06:25:27.189ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:35343/vpc-routes
10882025-04-30T06:25:27.189ZTRACtest_replacement_sanity (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
10892025-04-30T06:25:27.189ZINFOtest_replacement_sanity (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
10902025-04-30T06:25:27.190ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 1.88229017s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
10912025-04-30T06:25:27.190ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10922025-04-30T06:25:27.194ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
10932025-04-30T06:25:27.194ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10942025-04-30T06:25:27.194ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10952025-04-30T06:25:27.195ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10962025-04-30T06:25:27.204ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmppFz9MU/artifact-config.json server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
10972025-04-30T06:25:27.204ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmppFz9MU/artifact-config.json server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
10982025-04-30T06:25:27.204ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpWP18tA/artifact-config.json server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
10992025-04-30T06:25:27.204ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpWP18tA/artifact-config.json server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
11002025-04-30T06:25:27.301ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:47351 method = GET path = /artifact/sha256/{sha256} server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11012025-04-30T06:25:27.301ZINFOtest_replacement_sanity (dropshot (Repo Depot)): listening local_addr = [::1]:47351 server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
11022025-04-30T06:25:27.302ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:47351 server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
11032025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /artifacts sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11042025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /artifacts/{sha256} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11052025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11062025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /artifacts-config sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11072025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /artifacts-config sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11082025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11092025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11102025-04-30T06:25:27.329ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11112025-04-30T06:25:27.331ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /bootstore/status sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11122025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /datasets sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11132025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /disks/{disk_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11142025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /disks/{disk_id}/poke sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11152025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /eip-gateways sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11162025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /inventory sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11172025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /network-bootstore-config sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11182025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /network-bootstore-config sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11192025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /omicron-config sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11202025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /omicron-physical-disks sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11212025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /sled-identifiers sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11222025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /sled-role sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11232025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /sleds sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11242025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/dladm-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11252025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/ipadm-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11262025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/logs/download/{zone} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11272025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/logs/zones sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11282025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/nvmeadm-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11292025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/pargs-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11302025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/pfiles-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11312025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/pstack-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11322025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/zfs-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11332025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/zoneadm-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11342025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support/zpool-info sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11352025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11362025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11372025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11382025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11392025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11402025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11412025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11422025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11432025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11442025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /switch-ports sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11452025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /timesync sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11462025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = DELETE path = /v2p sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11472025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /v2p sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11482025-04-30T06:25:27.332ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /v2p sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11492025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = DELETE path = /vmms/{propolis_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11502025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /vmms/{propolis_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11512025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11522025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11532025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11542025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /vmms/{propolis_id}/poke sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11552025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11562025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11572025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /vmms/{propolis_id}/state sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11582025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /vmms/{propolis_id}/state sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11592025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11602025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /vpc-routes sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11612025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /vpc-routes sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11622025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zones sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11632025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = POST path = /zones/bundle-cleanup sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11642025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zones/bundle-cleanup/context sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11652025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = PUT path = /zones/bundle-cleanup/context sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11662025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zones/bundle-cleanup/utilization sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11672025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zones/bundles sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11682025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zones/bundles/{zone_name} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11692025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11702025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11712025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:42668 method = GET path = /zpools sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
11722025-04-30T06:25:27.333ZINFOtest_replacement_sanity (dropshot): listening local_addr = [::1]:42668 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
11732025-04-30T06:25:27.333ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:42668 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
11742025-04-30T06:25:27.333ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:48005 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11752025-04-30T06:25:27.338ZDEBGtest_replacement_sanity (omicron_sled_agent::sim::Server): contacting server nexus sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
11762025-04-30T06:25:27.338ZDEBGtest_replacement_sanity (NexusClient): client request body = Some(Body) method = POST sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = http://127.0.0.1:43527/sled-agents/039be560-54cc-49e3-88df-1a29dadbf913
11772025-04-30T06:25:27.352ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11782025-04-30T06:25:27.356ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11792025-04-30T06:25:27.356ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11802025-04-30T06:25:27.356ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11812025-04-30T06:25:27.357ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11822025-04-30T06:25:27.357ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11832025-04-30T06:25:27.358ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11842025-04-30T06:25:27.358ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11852025-04-30T06:25:27.358ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11862025-04-30T06:25:27.358ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11872025-04-30T06:25:27.360ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11882025-04-30T06:25:27.360ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11892025-04-30T06:25:27.361ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11902025-04-30T06:25:27.361ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11912025-04-30T06:25:27.364ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11922025-04-30T06:25:27.364ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11932025-04-30T06:25:27.365ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11942025-04-30T06:25:27.368ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11952025-04-30T06:25:27.368ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11962025-04-30T06:25:27.368ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11972025-04-30T06:25:27.368ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11982025-04-30T06:25:27.368ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11992025-04-30T06:25:27.369ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12002025-04-30T06:25:27.369ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:40388
12012025-04-30T06:25:27.370ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:48005 req_id = 20f43527-7fc3-47fc-a04f-eeebe6204a3e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
12022025-04-30T06:25:27.371ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1110 local_addr = [::1]:35343 method = GET remote_addr = [::1]:48005 req_id = 20f43527-7fc3-47fc-a04f-eeebe6204a3e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
12032025-04-30T06:25:27.371ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
12042025-04-30T06:25:27.371ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12052025-04-30T06:25:27.371ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12062025-04-30T06:25:27.371ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12072025-04-30T06:25:27.373ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12082025-04-30T06:25:27.374ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12092025-04-30T06:25:27.393ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12102025-04-30T06:25:27.393ZTRACtest_replacement_sanity (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
12112025-04-30T06:25:27.394ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12122025-04-30T06:25:27.394ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12132025-04-30T06:25:27.394ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12142025-04-30T06:25:27.394ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12152025-04-30T06:25:27.394ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12162025-04-30T06:25:27.406ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12172025-04-30T06:25:27.412ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
12182025-04-30T06:25:27.412ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
12192025-04-30T06:25:27.412ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
12202025-04-30T06:25:27.412ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
12212025-04-30T06:25:27.417ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12222025-04-30T06:25:27.450ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
12232025-04-30T06:25:27.451ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
12242025-04-30T06:25:27.451ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12252025-04-30T06:25:27.452ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12262025-04-30T06:25:27.453ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12272025-04-30T06:25:27.453ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12282025-04-30T06:25:27.453ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12292025-04-30T06:25:27.454ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12302025-04-30T06:25:27.460ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40388 req_id = 20e89e97-ed60-4e0b-99a2-66a71124d7af uri = /sled-agents/039be560-54cc-49e3-88df-1a29dadbf913
12312025-04-30T06:25:27.460ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 52 84 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 R............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51969 req_id = 9ff7873c-65c6-4bd2-88f8-e03eba824b8a
12322025-04-30T06:25:27.460ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21124,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51969 req_id = 9ff7873c-65c6-4bd2-88f8-e03eba824b8a
12332025-04-30T06:25:27.461ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21124, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51969 req_id = 9ff7873c-65c6-4bd2-88f8-e03eba824b8a
12342025-04-30T06:25:27.461ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:35343/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "20f43527-7fc3-47fc-a04f-eeebe6204a3e", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} })
12352025-04-30T06:25:27.461ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
12362025-04-30T06:25:27.462ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
12372025-04-30T06:25:27.462ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
12382025-04-30T06:25:27.463ZINFOtest_replacement_sanity (nexus): registered sled agent sled_uuid = 039be560-54cc-49e3-88df-1a29dadbf913
12392025-04-30T06:25:27.465ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:35343/artifacts-config
12402025-04-30T06:25:27.466ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12412025-04-30T06:25:27.466ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12422025-04-30T06:25:27.466ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12432025-04-30T06:25:27.467ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12442025-04-30T06:25:27.470ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:48239 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12452025-04-30T06:25:27.471ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
12462025-04-30T06:25:27.471ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 626.248921ms iteration = 2 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
12472025-04-30T06:25:27.474ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12482025-04-30T06:25:27.494ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12492025-04-30T06:25:27.494ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12502025-04-30T06:25:27.495ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12512025-04-30T06:25:27.495ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12522025-04-30T06:25:27.495ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:48239 req_id = 75f76946-755c-45e0-958f-d78dc806a070 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12532025-04-30T06:25:27.502ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpuK8mDj/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12542025-04-30T06:25:27.502ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpuK8mDj/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12552025-04-30T06:25:27.502ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpoFsHo0/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12562025-04-30T06:25:27.502ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpoFsHo0/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12572025-04-30T06:25:27.503ZINFOtest_replacement_sanity (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpuK8mDj/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12582025-04-30T06:25:27.504ZINFOtest_replacement_sanity (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpoFsHo0/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12592025-04-30T06:25:27.506ZINFOtest_replacement_sanity (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12602025-04-30T06:25:27.507ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 11510 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:48239 req_id = 75f76946-755c-45e0-958f-d78dc806a070 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12612025-04-30T06:25:27.507ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12622025-04-30T06:25:27.507ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12632025-04-30T06:25:27.508ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12642025-04-30T06:25:27.537ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:35343/artifacts-config", status: 204, headers: {"x-request-id": "75f76946-755c-45e0-958f-d78dc806a070", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} })
12652025-04-30T06:25:27.537ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:35343/artifacts
12662025-04-30T06:25:27.538ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:48239 req_id = d79511da-3556-410d-8d6a-63e4bb2baf3e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12672025-04-30T06:25:27.552ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 14064 local_addr = [::1]:35343 method = GET remote_addr = [::1]:48239 req_id = d79511da-3556-410d-8d6a-63e4bb2baf3e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12682025-04-30T06:25:27.566ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:35343/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "d79511da-3556-410d-8d6a-63e4bb2baf3e", "content-length": "26", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} })
12692025-04-30T06:25:27.566ZINFOtest_replacement_sanity (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:35343
12702025-04-30T06:25:27.568ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 2.250398422s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:25:27.537486739Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:25:27.566463674Z"}]}
12712025-04-30T06:25:27.579ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12722025-04-30T06:25:27.580ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12732025-04-30T06:25:27.582ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12742025-04-30T06:25:27.657ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:35343/v2p
12752025-04-30T06:25:27.658ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
12762025-04-30T06:25:27.658ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
12772025-04-30T06:25:27.658ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12782025-04-30T06:25:27.658ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
12792025-04-30T06:25:27.658ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
12802025-04-30T06:25:27.659ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
12812025-04-30T06:25:27.659ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 199488 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40388 req_id = 20e89e97-ed60-4e0b-99a2-66a71124d7af response_code = 204 uri = /sled-agents/039be560-54cc-49e3-88df-1a29dadbf913
12822025-04-30T06:25:27.660ZINFOtest_replacement_sanity (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
12832025-04-30T06:25:27.660ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:35343/eip-gateways
12842025-04-30T06:25:27.661ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12852025-04-30T06:25:27.661ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12862025-04-30T06:25:27.663ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12872025-04-30T06:25:27.691ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:42063 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12882025-04-30T06:25:27.691ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:48005 req_id = 284d4ada-6c56-4c2e-ac7e-d79fd5bc4b11 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12892025-04-30T06:25:27.695ZDEBGtest_replacement_sanity (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:43527/sled-agents/039be560-54cc-49e3-88df-1a29dadbf913", status: 204, headers: {"x-request-id": "20e89e97-ed60-4e0b-99a2-66a71124d7af", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} }) sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
12902025-04-30T06:25:27.695ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 4276 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:48005 req_id = 284d4ada-6c56-4c2e-ac7e-d79fd5bc4b11 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12912025-04-30T06:25:27.696ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12922025-04-30T06:25:27.699ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
12932025-04-30T06:25:27.699ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
12942025-04-30T06:25:27.699ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
12952025-04-30T06:25:27.699ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
12962025-04-30T06:25:27.700ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12972025-04-30T06:25:27.700ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12982025-04-30T06:25:27.700ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12992025-04-30T06:25:27.700ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13002025-04-30T06:25:27.700ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13012025-04-30T06:25:27.702ZDEBGtest_replacement_sanity (init_with_steps): Running step add_extra_sled_agent
13022025-04-30T06:25:27.702ZINFOtest_replacement_sanity (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13032025-04-30T06:25:27.768ZINFOtest_replacement_sanity (SledAgent): created simulated sled agent server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sim_mode = Explicit sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13042025-04-30T06:25:27.769ZINFOtest_replacement_sanity (omicron_sled_agent::sim::SimulatedUpstairs): registered sled 2823225f-fa9c-49ed-b0f5-744d993ad37d storage
13052025-04-30T06:25:27.770ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:42063 req_id = 0acc450f-ad6c-4bf4-8252-70c2ba9f729d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
13062025-04-30T06:25:27.770ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
13072025-04-30T06:25:27.770ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
13082025-04-30T06:25:27.770ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:35343/eip-gateways", status: 204, headers: {"x-request-id": "284d4ada-6c56-4c2e-ac7e-d79fd5bc4b11", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} })
13092025-04-30T06:25:27.772ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 2.462215784s iteration = 1 status = {}
13102025-04-30T06:25:27.772ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2349 local_addr = [::1]:35343 method = GET remote_addr = [::1]:42063 req_id = 0acc450f-ad6c-4bf4-8252-70c2ba9f729d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
13112025-04-30T06:25:27.774ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13122025-04-30T06:25:27.774ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13132025-04-30T06:25:27.774ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13142025-04-30T06:25:27.776ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13152025-04-30T06:25:27.783ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:35343/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "0acc450f-ad6c-4bf4-8252-70c2ba9f729d", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:26 GMT"} })
13162025-04-30T06:25:27.783ZINFOtest_replacement_sanity (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
13172025-04-30T06:25:27.784ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
13182025-04-30T06:25:27.784ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
13192025-04-30T06:25:27.785ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 2.484861545s iteration = 1 status = {}
13202025-04-30T06:25:27.788ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpIcxehd/artifact-config.json server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13212025-04-30T06:25:27.788ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpIcxehd/artifact-config.json server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13222025-04-30T06:25:27.788ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpk7WXqJ/artifact-config.json server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13232025-04-30T06:25:27.788ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpk7WXqJ/artifact-config.json server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13242025-04-30T06:25:27.843ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:61031 method = GET path = /artifact/sha256/{sha256} server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13252025-04-30T06:25:27.846ZINFOtest_replacement_sanity (dropshot (Repo Depot)): listening local_addr = [::1]:61031 server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13262025-04-30T06:25:27.847ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:61031 server = 2823225f-fa9c-49ed-b0f5-744d993ad37d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13272025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /artifacts sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13282025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /artifacts/{sha256} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13292025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13302025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /artifacts-config sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13312025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /artifacts-config sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13322025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13332025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13342025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13352025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /bootstore/status sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13362025-04-30T06:25:27.851ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /datasets sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13372025-04-30T06:25:27.852ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /disks/{disk_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13382025-04-30T06:25:27.852ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /disks/{disk_id}/poke sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13392025-04-30T06:25:27.852ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /eip-gateways sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13402025-04-30T06:25:27.852ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /inventory sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13412025-04-30T06:25:27.857ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /network-bootstore-config sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13422025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /network-bootstore-config sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13432025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /omicron-config sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13442025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /omicron-physical-disks sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13452025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /sled-identifiers sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13462025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /sled-role sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13472025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /sleds sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13482025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/dladm-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13492025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/ipadm-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13502025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/logs/download/{zone} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13512025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/logs/zones sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13522025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/nvmeadm-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13532025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/pargs-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13542025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/pfiles-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13552025-04-30T06:25:27.858ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/pstack-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13562025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/zfs-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13572025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/zoneadm-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13582025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support/zpool-info sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13592025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13602025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13612025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13622025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13632025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13642025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13652025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13662025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13672025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13682025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /switch-ports sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13692025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /timesync sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13702025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = DELETE path = /v2p sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13712025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /v2p sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13722025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /v2p sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13732025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = DELETE path = /vmms/{propolis_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13742025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /vmms/{propolis_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13752025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13762025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13772025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13782025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /vmms/{propolis_id}/poke sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13792025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13802025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13812025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /vmms/{propolis_id}/state sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13822025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /vmms/{propolis_id}/state sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13832025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13842025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /vpc-routes sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13852025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /vpc-routes sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13862025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zones sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13872025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = POST path = /zones/bundle-cleanup sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13882025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zones/bundle-cleanup/context sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13892025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = PUT path = /zones/bundle-cleanup/context sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13902025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zones/bundle-cleanup/utilization sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13912025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zones/bundles sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13922025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zones/bundles/{zone_name} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13932025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13942025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13952025-04-30T06:25:27.859ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:47918 method = GET path = /zpools sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d versions = all
13962025-04-30T06:25:27.859ZINFOtest_replacement_sanity (dropshot): listening local_addr = [::1]:47918 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13972025-04-30T06:25:27.860ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:47918 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13982025-04-30T06:25:27.861ZDEBGtest_replacement_sanity (omicron_sled_agent::sim::Server): contacting server nexus sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
13992025-04-30T06:25:27.861ZDEBGtest_replacement_sanity (NexusClient): client request body = Some(Body) method = POST sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = http://127.0.0.1:43527/sled-agents/2823225f-fa9c-49ed-b0f5-744d993ad37d
14002025-04-30T06:25:27.861ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 41 3a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 A:..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49449 req_id = c72166ef-9824-4856-a637-9c7ac2504e28
14012025-04-30T06:25:27.862ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16698,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49449 req_id = c72166ef-9824-4856-a637-9c7ac2504e28
14022025-04-30T06:25:27.862ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 16698, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49449 req_id = c72166ef-9824-4856-a637-9c7ac2504e28
14032025-04-30T06:25:27.862ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14042025-04-30T06:25:27.862ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14052025-04-30T06:25:27.863ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14062025-04-30T06:25:27.863ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14072025-04-30T06:25:27.866ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:48734
14082025-04-30T06:25:27.866ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
14092025-04-30T06:25:27.866ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 208.094825ms iteration = 3 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
14102025-04-30T06:25:27.878ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14112025-04-30T06:25:27.885ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
14122025-04-30T06:25:27.885ZTRACtest_replacement_sanity (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
14132025-04-30T06:25:27.885ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14142025-04-30T06:25:27.885ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14152025-04-30T06:25:27.886ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14162025-04-30T06:25:27.890ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:48734 req_id = 4dc59e32-32c6-40bc-bbe2-18f7e193d175 uri = /sled-agents/2823225f-fa9c-49ed-b0f5-744d993ad37d
14172025-04-30T06:25:27.891ZINFOtest_replacement_sanity (nexus): registered sled agent sled_uuid = 2823225f-fa9c-49ed-b0f5-744d993ad37d
14182025-04-30T06:25:27.893ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14192025-04-30T06:25:27.893ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14202025-04-30T06:25:27.894ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14212025-04-30T06:25:27.898ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14222025-04-30T06:25:27.898ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14232025-04-30T06:25:27.899ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14242025-04-30T06:25:27.900ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
14252025-04-30T06:25:27.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
14262025-04-30T06:25:27.900ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14272025-04-30T06:25:27.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
14282025-04-30T06:25:27.900ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
14292025-04-30T06:25:27.900ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
14302025-04-30T06:25:27.900ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 10074 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:48734 req_id = 4dc59e32-32c6-40bc-bbe2-18f7e193d175 response_code = 204 uri = /sled-agents/2823225f-fa9c-49ed-b0f5-744d993ad37d
14312025-04-30T06:25:27.905ZDEBGtest_replacement_sanity (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:43527/sled-agents/2823225f-fa9c-49ed-b0f5-744d993ad37d", status: 204, headers: {"x-request-id": "4dc59e32-32c6-40bc-bbe2-18f7e193d175", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} }) sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
14322025-04-30T06:25:27.905ZDEBGtest_replacement_sanity (init_with_steps): Running step add_extra_sled_agent
14332025-04-30T06:25:27.905ZINFOtest_replacement_sanity (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14342025-04-30T06:25:27.951ZINFOtest_replacement_sanity (SledAgent): created simulated sled agent server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sim_mode = Explicit sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14352025-04-30T06:25:27.951ZINFOtest_replacement_sanity (omicron_sled_agent::sim::SimulatedUpstairs): registered sled 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da storage
14362025-04-30T06:25:27.951ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14372025-04-30T06:25:27.951ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14382025-04-30T06:25:27.952ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14392025-04-30T06:25:27.953ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14402025-04-30T06:25:27.960ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
14412025-04-30T06:25:27.960ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
14422025-04-30T06:25:27.960ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
14432025-04-30T06:25:27.960ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
14442025-04-30T06:25:27.972ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14452025-04-30T06:25:27.973ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14462025-04-30T06:25:27.973ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14472025-04-30T06:25:27.973ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14482025-04-30T06:25:27.975ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14492025-04-30T06:25:27.976ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
14502025-04-30T06:25:27.976ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
14512025-04-30T06:25:27.977ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpTd5rbA/artifact-config.json server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14522025-04-30T06:25:27.977ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpTd5rbA/artifact-config.json server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14532025-04-30T06:25:27.977ZINFOtest_replacement_sanity (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpCidRqr/artifact-config.json server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14542025-04-30T06:25:27.977ZDEBGtest_replacement_sanity (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpCidRqr/artifact-config.json server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14552025-04-30T06:25:28.035ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:64698 method = GET path = /artifact/sha256/{sha256} server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14562025-04-30T06:25:28.035ZINFOtest_replacement_sanity (dropshot (Repo Depot)): listening local_addr = [::1]:64698 server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14572025-04-30T06:25:28.035ZDEBGtest_replacement_sanity (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:64698 server = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
14582025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /artifacts sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14592025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /artifacts/{sha256} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14602025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14612025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /artifacts-config sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14622025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /artifacts-config sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14632025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14642025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14652025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14662025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /bootstore/status sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14672025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /datasets sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14682025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /disks/{disk_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14692025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /disks/{disk_id}/poke sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14702025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /eip-gateways sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14712025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /inventory sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14722025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /network-bootstore-config sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14732025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /network-bootstore-config sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14742025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /omicron-config sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14752025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /omicron-physical-disks sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14762025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /sled-identifiers sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14772025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /sled-role sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14782025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /sleds sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14792025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/dladm-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14802025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/ipadm-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14812025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/logs/download/{zone} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14822025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/logs/zones sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14832025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/nvmeadm-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14842025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/pargs-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14852025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/pfiles-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14862025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/pstack-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14872025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/zfs-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14882025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/zoneadm-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14892025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support/zpool-info sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14902025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14912025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14922025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14932025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14942025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14952025-04-30T06:25:28.040ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14962025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14972025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14982025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
14992025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /switch-ports sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15002025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /timesync sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15012025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = DELETE path = /v2p sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15022025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /v2p sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15032025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /v2p sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15042025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = DELETE path = /vmms/{propolis_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15052025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /vmms/{propolis_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15062025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15072025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15082025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15092025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /vmms/{propolis_id}/poke sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15102025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15112025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15122025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /vmms/{propolis_id}/state sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15132025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /vmms/{propolis_id}/state sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15142025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15152025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /vpc-routes sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15162025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /vpc-routes sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15172025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zones sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15182025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = POST path = /zones/bundle-cleanup sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15192025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zones/bundle-cleanup/context sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15202025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = PUT path = /zones/bundle-cleanup/context sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15212025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zones/bundle-cleanup/utilization sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15222025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zones/bundles sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15232025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zones/bundles/{zone_name} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15242025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15252025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15262025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:35744 method = GET path = /zpools sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da versions = all
15272025-04-30T06:25:28.041ZINFOtest_replacement_sanity (dropshot): listening local_addr = [::1]:35744 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15282025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:35744 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15292025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (omicron_sled_agent::sim::Server): contacting server nexus sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15302025-04-30T06:25:28.041ZDEBGtest_replacement_sanity (NexusClient): client request body = Some(Body) method = POST sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = http://127.0.0.1:43527/sled-agents/98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15312025-04-30T06:25:28.041ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3c c7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 <............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62796 req_id = c2a84e57-6769-446d-880c-032be746e7c0
15322025-04-30T06:25:28.042ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15559,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62796 req_id = c2a84e57-6769-446d-880c-032be746e7c0
15332025-04-30T06:25:28.042ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15559, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62796 req_id = c2a84e57-6769-446d-880c-032be746e7c0
15342025-04-30T06:25:28.042ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:49169
15352025-04-30T06:25:28.043ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
15362025-04-30T06:25:28.043ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 143.170177ms iteration = 4 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
15372025-04-30T06:25:28.043ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:49169 req_id = fa96d354-a840-4d15-abcf-8431ac03424e uri = /sled-agents/98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15382025-04-30T06:25:28.044ZINFOtest_replacement_sanity (nexus): registered sled agent sled_uuid = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15392025-04-30T06:25:28.049ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
15402025-04-30T06:25:28.049ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15412025-04-30T06:25:28.049ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15422025-04-30T06:25:28.050ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15432025-04-30T06:25:28.057ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15442025-04-30T06:25:28.057ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15452025-04-30T06:25:28.057ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15462025-04-30T06:25:28.058ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15472025-04-30T06:25:28.069ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15482025-04-30T06:25:28.076ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
15492025-04-30T06:25:28.076ZTRACtest_replacement_sanity (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
15502025-04-30T06:25:28.076ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
15512025-04-30T06:25:28.076ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
15522025-04-30T06:25:28.076ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15532025-04-30T06:25:28.076ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15542025-04-30T06:25:28.076ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15552025-04-30T06:25:28.081ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
15562025-04-30T06:25:28.081ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15572025-04-30T06:25:28.081ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15582025-04-30T06:25:28.081ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15592025-04-30T06:25:28.121ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15602025-04-30T06:25:28.122ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15612025-04-30T06:25:28.122ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15622025-04-30T06:25:28.122ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15632025-04-30T06:25:28.141ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15642025-04-30T06:25:28.151ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
15652025-04-30T06:25:28.151ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15662025-04-30T06:25:28.151ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15672025-04-30T06:25:28.152ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15682025-04-30T06:25:28.163ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15692025-04-30T06:25:28.164ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15702025-04-30T06:25:28.164ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15712025-04-30T06:25:28.177ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
15722025-04-30T06:25:28.177ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
15732025-04-30T06:25:28.178ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
15742025-04-30T06:25:28.178ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
15752025-04-30T06:25:28.178ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15762025-04-30T06:25:28.178ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15772025-04-30T06:25:28.178ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15782025-04-30T06:25:28.178ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15792025-04-30T06:25:28.179ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15802025-04-30T06:25:28.179ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 135444 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:49169 req_id = fa96d354-a840-4d15-abcf-8431ac03424e response_code = 204 uri = /sled-agents/98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15812025-04-30T06:25:28.181ZDEBGtest_replacement_sanity (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:43527/sled-agents/98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da", status: 204, headers: {"x-request-id": "fa96d354-a840-4d15-abcf-8431ac03424e", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} }) sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
15822025-04-30T06:25:28.182ZDEBGtest_replacement_sanity (init_with_steps): Running step start_crucible_pantry
15832025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15842025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15852025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15862025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15872025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15882025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15892025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15902025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15912025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15922025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15932025-04-30T06:25:28.183ZDEBGtest_replacement_sanity (dropshot): registered endpoint kind = pantry local_addr = [::1]:59920 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
15942025-04-30T06:25:28.183ZINFOtest_replacement_sanity (dropshot): listening kind = pantry local_addr = [::1]:59920 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15952025-04-30T06:25:28.184ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:59920 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15962025-04-30T06:25:28.184ZINFOtest_replacement_sanity (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:59920 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15972025-04-30T06:25:28.184ZDEBGtest_replacement_sanity (init_with_steps): Running step populate_internal_dns
15982025-04-30T06:25:28.184ZDEBGtest_replacement_sanity: Populating Internal DNS
15992025-04-30T06:25:28.244ZINFOtest_replacement_sanity: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:25:28.244128016Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 40918, target: "dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal", }, ), Srv( Srv { prio: 0, weight: 0, port: 50113, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 32936, target: "dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal", }, ), Srv( Srv { prio: 0, weight: 0, port: 53480, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "3ae50786-e3ad-43a3-866e-b9ad16af275d.host": [ Aaaa( ::1, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 59920, target: "774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal", }, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host": [ Aaaa( ::1, ), ], "b3328697-4f10-488d-bada-1e267a8b2fdd.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 47351, target: "039be560-54cc-49e3-88df-1a29dadbf913.sled.control-plane.oxide.internal", }, ), Srv( Srv { prio: 0, weight: 0, port: 46651, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 35427, target: "3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 43527, target: "c31da0ae-930d-4dd9-a897-159d5a6cd420.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 64470, target: "9dd2420b-e2ca-4ec4-9b23-91c5a9cc3e5f.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "9dd2420b-e2ca-4ec4-9b23-91c5a9cc3e5f.host.control-plane.oxide.internal", }, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "9dd2420b-e2ca-4ec4-9b23-91c5a9cc3e5f.host.control-plane.oxide.internal", }, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 34917, target: "a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal", }, ), ], "c31da0ae-930d-4dd9-a897-159d5a6cd420.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 44212, target: "dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal", }, ), Srv( Srv { prio: 0, weight: 0, port: 35157, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "039be560-54cc-49e3-88df-1a29dadbf913.sled": [ Aaaa( ::1, ), ], "9dd2420b-e2ca-4ec4-9b23-91c5a9cc3e5f.host": [ Aaaa( ::1, ), ], "a4736702-dd27-45ae-aa41-29a35dcc68a1.host": [ Aaaa( ::1, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "9dd2420b-e2ca-4ec4-9b23-91c5a9cc3e5f.host.control-plane.oxide.internal", }, ), ], "774760d4-5767-4f71-9092-86a22ff16c67.host": [ Aaaa( ::1, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 46323, target: "b3328697-4f10-488d-bada-1e267a8b2fdd.host.control-plane.oxide.internal", }, ), ], }, }, ], }
16002025-04-30T06:25:28.244ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:34917/config
16012025-04-30T06:25:28.252ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16022025-04-30T06:25:28.252ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16032025-04-30T06:25:28.258ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16042025-04-30T06:25:28.258ZINFOtest_replacement_sanity (http): accepted connection kind = dns local_addr = [::1]:34917 remote_addr = [::1]:38146
16052025-04-30T06:25:28.258ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16062025-04-30T06:25:28.265ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
16072025-04-30T06:25:28.265ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
16082025-04-30T06:25:28.265ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
16092025-04-30T06:25:28.266ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
16102025-04-30T06:25:28.272ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16112025-04-30T06:25:28.272ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16122025-04-30T06:25:28.273ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16132025-04-30T06:25:28.274ZTRACtest_replacement_sanity (http): incoming request kind = dns local_addr = [::1]:34917 method = PUT remote_addr = [::1]:38146 req_id = 3d490492-b870-4f75-82e3-dbb8abbc611b uri = /config
16142025-04-30T06:25:28.275ZINFOtest_replacement_sanity (store): attempting generation update new_generation = 1 req_id = 3d490492-b870-4f75-82e3-dbb8abbc611b
16152025-04-30T06:25:28.275ZINFOtest_replacement_sanity (store): pruning trees for generations newer than 0
16162025-04-30T06:25:28.275ZDEBGtest_replacement_sanity (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
16172025-04-30T06:25:28.276ZDEBGtest_replacement_sanity (store): updating current config
16182025-04-30T06:25:28.276ZDEBGtest_replacement_sanity (store): flushing default tree
16192025-04-30T06:25:28.277ZINFOtest_replacement_sanity (store): pruning trees for generations older than 1 keep = 3
16202025-04-30T06:25:28.277ZINFOtest_replacement_sanity (store): updated generation new_generation = 1 req_id = 3d490492-b870-4f75-82e3-dbb8abbc611b
16212025-04-30T06:25:28.277ZINFOtest_replacement_sanity (http): request completed kind = dns latency_us = 2621 local_addr = [::1]:34917 method = PUT remote_addr = [::1]:38146 req_id = 3d490492-b870-4f75-82e3-dbb8abbc611b response_code = 204 uri = /config
16222025-04-30T06:25:28.277ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
16232025-04-30T06:25:28.277ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
16242025-04-30T06:25:28.281ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
16252025-04-30T06:25:28.281ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16262025-04-30T06:25:28.281ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16272025-04-30T06:25:28.281ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16282025-04-30T06:25:28.281ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16292025-04-30T06:25:28.282ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:34917/config", status: 204, headers: {"x-request-id": "3d490492-b870-4f75-82e3-dbb8abbc611b", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
16302025-04-30T06:25:28.284ZDEBGtest_replacement_sanity (init_with_steps): Running step configure_sled_agents
16312025-04-30T06:25:28.336ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:35343/omicron-config
16322025-04-30T06:25:28.337ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 87 c1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51076 req_id = 4358c78f-1ce9-4127-b5ff-aea1f3fb65fc
16332025-04-30T06:25:28.338ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34753,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51076 req_id = 4358c78f-1ce9-4127-b5ff-aea1f3fb65fc
16342025-04-30T06:25:28.338ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
16352025-04-30T06:25:28.338ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
16362025-04-30T06:25:28.338ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
16372025-04-30T06:25:28.338ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51076 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4358c78f-1ce9-4127-b5ff-aea1f3fb65fc
16382025-04-30T06:25:28.340ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:56197 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
16392025-04-30T06:25:28.341ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3193, tv_nsec: 140828503 } })
16402025-04-30T06:25:28.342ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
16412025-04-30T06:25:28.342ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
16422025-04-30T06:25:28.344ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16432025-04-30T06:25:28.344ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16442025-04-30T06:25:28.344ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16452025-04-30T06:25:28.346ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16462025-04-30T06:25:28.350ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:56197 req_id = 3c2aee96-67f8-43aa-9644-307c440ff79f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
16472025-04-30T06:25:28.350ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16482025-04-30T06:25:28.350ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16492025-04-30T06:25:28.351ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16502025-04-30T06:25:28.351ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16512025-04-30T06:25:28.353ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: af 33 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .3..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58899 req_id = f7de07ce-c2e6-4029-b569-bcbddfea046b
16522025-04-30T06:25:28.353ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44851,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58899 req_id = f7de07ce-c2e6-4029-b569-bcbddfea046b
16532025-04-30T06:25:28.353ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
16542025-04-30T06:25:28.354ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58899 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f7de07ce-c2e6-4029-b569-bcbddfea046b
16552025-04-30T06:25:28.354ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cc b2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57258 req_id = 39523c66-9eb4-4137-b323-99a78d5fc8a5
16562025-04-30T06:25:28.354ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52402,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57258 req_id = 39523c66-9eb4-4137-b323-99a78d5fc8a5
16572025-04-30T06:25:28.354ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
16582025-04-30T06:25:28.354ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57258 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 39523c66-9eb4-4137-b323-99a78d5fc8a5
16592025-04-30T06:25:28.358ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 7734 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:56197 req_id = 3c2aee96-67f8-43aa-9644-307c440ff79f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
16602025-04-30T06:25:28.480ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
16612025-04-30T06:25:28.481ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:35343/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c2aee96-67f8-43aa-9644-307c440ff79f", "content-length": "26", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
16622025-04-30T06:25:28.482ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:35343/network-bootstore-config
16632025-04-30T06:25:28.486ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:56197 req_id = 81889e52-8390-4093-9f32-a10487ad095d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
16642025-04-30T06:25:28.486ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ad b6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63290 req_id = 1689b7b1-a6a4-4c2d-a7b0-566f5e6c7ebb
16652025-04-30T06:25:28.487ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44470,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63290 req_id = 1689b7b1-a6a4-4c2d-a7b0-566f5e6c7ebb
16662025-04-30T06:25:28.487ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
16672025-04-30T06:25:28.487ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44470, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63290 req_id = 1689b7b1-a6a4-4c2d-a7b0-566f5e6c7ebb
16682025-04-30T06:25:28.488ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1961 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:56197 req_id = 81889e52-8390-4093-9f32-a10487ad095d response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
16692025-04-30T06:25:28.521ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16702025-04-30T06:25:28.536ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
16712025-04-30T06:25:28.536ZTRACtest_replacement_sanity (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
16722025-04-30T06:25:28.536ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
16732025-04-30T06:25:28.536ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16742025-04-30T06:25:28.536ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16752025-04-30T06:25:28.536ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16762025-04-30T06:25:28.537ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16772025-04-30T06:25:28.540ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
16782025-04-30T06:25:28.540ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
16792025-04-30T06:25:28.540ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
16802025-04-30T06:25:28.542ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:35343/network-bootstore-config", status: 204, headers: {"x-request-id": "81889e52-8390-4093-9f32-a10487ad095d", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
16812025-04-30T06:25:28.627ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:42668/omicron-config
16822025-04-30T06:25:28.629ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16832025-04-30T06:25:28.629ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16842025-04-30T06:25:28.629ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16852025-04-30T06:25:28.629ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16862025-04-30T06:25:28.630ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:56940 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
16872025-04-30T06:25:28.639ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = PUT remote_addr = [::1]:56940 req_id = 37abf38f-b6e6-4cf0-ad09-cf0c7a72326b sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /omicron-config
16882025-04-30T06:25:28.644ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 5268 local_addr = [::1]:42668 method = PUT remote_addr = [::1]:56940 req_id = 37abf38f-b6e6-4cf0-ad09-cf0c7a72326b response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /omicron-config
16892025-04-30T06:25:28.645ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16902025-04-30T06:25:28.645ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
16912025-04-30T06:25:28.645ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
16922025-04-30T06:25:28.646ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
16932025-04-30T06:25:28.648ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:42668/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "37abf38f-b6e6-4cf0-ad09-cf0c7a72326b", "content-length": "26", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
16942025-04-30T06:25:28.648ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:42668/network-bootstore-config
16952025-04-30T06:25:28.655ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = PUT remote_addr = [::1]:56940 req_id = 357c183d-ce62-4cfa-adb3-b81682913559 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /network-bootstore-config
16962025-04-30T06:25:28.655ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0f4cc7e-c4ab-4a6c-93fa-9b09d0ed24c6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
16972025-04-30T06:25:28.656ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
16982025-04-30T06:25:28.656ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1815 local_addr = [::1]:42668 method = PUT remote_addr = [::1]:56940 req_id = 357c183d-ce62-4cfa-adb3-b81682913559 response_code = 204 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /network-bootstore-config
16992025-04-30T06:25:28.657ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17002025-04-30T06:25:28.670ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17012025-04-30T06:25:28.671ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
17022025-04-30T06:25:28.671ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
17032025-04-30T06:25:28.671ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
17042025-04-30T06:25:28.674ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:42668/network-bootstore-config", status: 204, headers: {"x-request-id": "357c183d-ce62-4cfa-adb3-b81682913559", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
17052025-04-30T06:25:28.740ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:47918/omicron-config
17062025-04-30T06:25:28.745ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:42686 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
17072025-04-30T06:25:28.753ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = PUT remote_addr = [::1]:42686 req_id = af3ab6c7-9587-453a-ade3-be738d9209ce sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /omicron-config
17082025-04-30T06:25:28.756ZDEBGtest_replacement_sanity (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
17092025-04-30T06:25:28.756ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
17102025-04-30T06:25:28.756ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17112025-04-30T06:25:28.756ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
17122025-04-30T06:25:28.756ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
17132025-04-30T06:25:28.757ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
17142025-04-30T06:25:28.757ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 4075 local_addr = [::1]:47918 method = PUT remote_addr = [::1]:42686 req_id = af3ab6c7-9587-453a-ade3-be738d9209ce response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /omicron-config
17152025-04-30T06:25:28.758ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "533398a8-81c3-418f-b76f-585fabf4101c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
17162025-04-30T06:25:28.759ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
17172025-04-30T06:25:28.760ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:47918/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "af3ab6c7-9587-453a-ade3-be738d9209ce", "content-length": "26", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
17182025-04-30T06:25:28.760ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:47918/network-bootstore-config
17192025-04-30T06:25:28.766ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17202025-04-30T06:25:28.774ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
17212025-04-30T06:25:28.774ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
17222025-04-30T06:25:28.775ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
17232025-04-30T06:25:28.775ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
17242025-04-30T06:25:28.778ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = PUT remote_addr = [::1]:42686 req_id = 9fe45d49-84b2-4b3d-9422-864383ace2b3 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /network-bootstore-config
17252025-04-30T06:25:28.779ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1118 local_addr = [::1]:47918 method = PUT remote_addr = [::1]:42686 req_id = 9fe45d49-84b2-4b3d-9422-864383ace2b3 response_code = 204 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /network-bootstore-config
17262025-04-30T06:25:28.790ZTRACtest_replacement_sanity (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
17272025-04-30T06:25:28.790ZDEBGtest_replacement_sanity (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
17282025-04-30T06:25:28.790ZDEBGtest_replacement_sanity (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
17292025-04-30T06:25:28.790ZDEBGtest_replacement_sanity (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
17302025-04-30T06:25:28.791ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:47918/network-bootstore-config", status: 204, headers: {"x-request-id": "9fe45d49-84b2-4b3d-9422-864383ace2b3", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
17312025-04-30T06:25:28.852ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:35744/omicron-config
17322025-04-30T06:25:28.856ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:52708 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
17332025-04-30T06:25:28.858ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = PUT remote_addr = [::1]:52708 req_id = f4b0e8d0-e16f-4da2-a2c6-c8c47b12adf8 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /omicron-config
17342025-04-30T06:25:28.859ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1184 local_addr = [::1]:35744 method = PUT remote_addr = [::1]:52708 req_id = f4b0e8d0-e16f-4da2-a2c6-c8c47b12adf8 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /omicron-config
17352025-04-30T06:25:28.859ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:35744/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4b0e8d0-e16f-4da2-a2c6-c8c47b12adf8", "content-length": "26", "date": "Wed, 30 Apr 2025 06:25:27 GMT"} })
17362025-04-30T06:25:28.859ZDEBGtest_replacement_sanity: client request body = Some(Body) method = PUT uri = http://[::1]:35744/network-bootstore-config
17372025-04-30T06:25:28.863ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = PUT remote_addr = [::1]:52708 req_id = 2319e1e0-c97a-4111-a71e-71c64f2f3416 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /network-bootstore-config
17382025-04-30T06:25:28.863ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2665 local_addr = [::1]:35744 method = PUT remote_addr = [::1]:52708 req_id = 2319e1e0-c97a-4111-a71e-71c64f2f3416 response_code = 204 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /network-bootstore-config
17392025-04-30T06:25:28.863ZDEBGtest_replacement_sanity: client response result = Ok(Response { url: "http://[::1]:35744/network-bootstore-config", status: 204, headers: {"x-request-id": "2319e1e0-c97a-4111-a71e-71c64f2f3416", "date": "Wed, 30 Apr 2025 06:25:28 GMT"} })
17402025-04-30T06:25:28.866ZDEBGtest_replacement_sanity (init_with_steps): Running step start_nexus_external
17412025-04-30T06:25:28.866ZDEBGtest_replacement_sanity: Starting Nexus (external API)
17422025-04-30T06:25:31.580ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
17432025-04-30T06:25:31.581ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17442025-04-30T06:25:31.581ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17452025-04-30T06:25:31.581ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17462025-04-30T06:25:31.581ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17472025-04-30T06:25:31.582ZDEBGtest_replacement_sanity (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
17482025-04-30T06:25:31.582ZDEBGtest_replacement_sanity (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
17492025-04-30T06:25:31.582ZINFOtest_replacement_sanity (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
17502025-04-30T06:25:31.582ZDEBGtest_replacement_sanity (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 6.273594236s
17512025-04-30T06:25:31.582ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 6.273870046s iteration = 1 status = {}
17522025-04-30T06:25:31.587ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97679d50-db14-47f7-ad98-6287dca923f0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
17532025-04-30T06:25:31.587ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
17542025-04-30T06:25:31.588ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17552025-04-30T06:25:31.592ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
17562025-04-30T06:25:31.592ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17572025-04-30T06:25:31.592ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17582025-04-30T06:25:31.592ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17592025-04-30T06:25:31.606ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17602025-04-30T06:25:31.606ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17612025-04-30T06:25:31.606ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17622025-04-30T06:25:31.631ZINFOtest_replacement_sanity (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
17632025-04-30T06:25:31.631ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17642025-04-30T06:25:31.632ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17652025-04-30T06:25:31.632ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17662025-04-30T06:25:31.639ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
17672025-04-30T06:25:31.639ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17682025-04-30T06:25:31.639ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17692025-04-30T06:25:31.639ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17702025-04-30T06:25:31.640ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17712025-04-30T06:25:31.660ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17722025-04-30T06:25:31.672ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
17732025-04-30T06:25:31.672ZDEBGtest_replacement_sanity (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
17742025-04-30T06:25:31.672ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:25:31.672ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:25:31.673ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:25:31.690ZINFOtest_replacement_sanity (InternalApi): Using static configuration for external switchports
17782025-04-30T06:25:31.690ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17792025-04-30T06:25:31.690ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17802025-04-30T06:25:31.690ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17812025-04-30T06:25:31.754ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7646af0b-55e5-431c-862d-739cedd58673", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
17822025-04-30T06:25:31.755ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
17832025-04-30T06:25:31.779ZINFOtest_replacement_sanity (InternalApi): Recording Rack Network Configuration
17842025-04-30T06:25:31.779ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17852025-04-30T06:25:31.779ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17862025-04-30T06:25:31.780ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17872025-04-30T06:25:31.884ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80ce5bf3-f8fe-4fd9-ad39-7d077ca5f0fe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
17882025-04-30T06:25:31.891ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
17892025-04-30T06:25:31.934ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df3950d7-2bec-4b7a-9e19-87acfed147a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
17902025-04-30T06:25:31.935ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
17912025-04-30T06:25:31.935ZTRACtest_replacement_sanity (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
17922025-04-30T06:25:31.935ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17932025-04-30T06:25:31.935ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17942025-04-30T06:25:31.935ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
17952025-04-30T06:25:31.936ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17962025-04-30T06:25:31.939ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1497d94-90ca-4811-8cf3-5902afac1f16", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
17972025-04-30T06:25:31.939ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
17982025-04-30T06:25:31.940ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17992025-04-30T06:25:31.943ZDEBGtest_replacement_sanity (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18002025-04-30T06:25:31.943ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:25:31.943ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:25:31.944ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:25:31.951ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 1c18de8a-4b68-42f1-b480-7ad3e3cff5ec, lookup_type: ByName("oxide-service-pool") }
18042025-04-30T06:25:31.951ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18052025-04-30T06:25:31.951ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18062025-04-30T06:25:31.951ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
18072025-04-30T06:25:31.952ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18082025-04-30T06:25:31.952ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01d2ca0f-faeb-4823-bc0c-9d2ba61b3c98", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18092025-04-30T06:25:31.952ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
18102025-04-30T06:25:31.954ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b61da49-ac0b-4b04-811e-e851d91cbb82", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18112025-04-30T06:25:31.954ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
18122025-04-30T06:25:31.954ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18132025-04-30T06:25:31.959ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 1c18de8a-4b68-42f1-b480-7ad3e3cff5ec, lookup_type: ByName("oxide-service-pool") } result = Ok(())
18142025-04-30T06:25:31.959ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18152025-04-30T06:25:31.959ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
18162025-04-30T06:25:31.960ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18172025-04-30T06:25:31.960ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3d2afb1-d4ae-4fce-829e-63b853a481ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18182025-04-30T06:25:31.961ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
18192025-04-30T06:25:31.963ZTRACtest_replacement_sanity (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 1c18de8a-4b68-42f1-b480-7ad3e3cff5ec, lookup_type: ByName("oxide-service-pool") }
18202025-04-30T06:25:31.963ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18212025-04-30T06:25:31.963ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18222025-04-30T06:25:31.963ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
18232025-04-30T06:25:31.964ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18242025-04-30T06:25:31.964ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5d21ef5-90b5-4f5f-b77d-89f5e62a3917", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18252025-04-30T06:25:31.964ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
18262025-04-30T06:25:31.966ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "caa403ca-eb79-443e-a1bf-b458e27d1c39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18272025-04-30T06:25:31.967ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
18282025-04-30T06:25:31.967ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18292025-04-30T06:25:31.970ZDEBGtest_replacement_sanity (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 1c18de8a-4b68-42f1-b480-7ad3e3cff5ec, lookup_type: ByName("oxide-service-pool") } result = Ok(())
18302025-04-30T06:25:31.972ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3896de8-4825-451e-be94-03df0d865c78", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18312025-04-30T06:25:31.972ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
18322025-04-30T06:25:31.974ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66c55435-fb28-4a67-80d3-fe5d07517a57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18332025-04-30T06:25:31.974ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
18342025-04-30T06:25:31.976ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "571b4bde-54f5-4b41-a4a3-844f72023663", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18352025-04-30T06:25:31.977ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
18362025-04-30T06:25:31.979ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d29fd9b7-1b64-4892-a203-dddd86d263a9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18372025-04-30T06:25:31.980ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
18382025-04-30T06:25:31.980ZTRACtest_replacement_sanity (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 1c18de8a-4b68-42f1-b480-7ad3e3cff5ec, lookup_type: ByName("oxide-service-pool") }
18392025-04-30T06:25:31.980ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18402025-04-30T06:25:31.980ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18412025-04-30T06:25:31.980ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
18422025-04-30T06:25:31.981ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18432025-04-30T06:25:31.983ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbf346ee-f70d-494a-8d80-060ffe06d65e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18442025-04-30T06:25:31.983ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
18452025-04-30T06:25:31.984ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18462025-04-30T06:25:31.989ZDEBGtest_replacement_sanity (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: 1c18de8a-4b68-42f1-b480-7ad3e3cff5ec, lookup_type: ByName("oxide-service-pool") } result = Ok(())
18472025-04-30T06:25:31.992ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67e2026-64dd-499c-affc-9f6633c14c06", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18482025-04-30T06:25:31.992ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
18492025-04-30T06:25:31.994ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65488262-841a-4058-bdd4-ca741fae077c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18502025-04-30T06:25:31.994ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
18512025-04-30T06:25:31.995ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
18522025-04-30T06:25:31.995ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18532025-04-30T06:25:31.995ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18542025-04-30T06:25:31.995ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
18552025-04-30T06:25:31.995ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18562025-04-30T06:25:32.005ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9996cc2-85de-4fd7-8cee-2e32e2a809c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18572025-04-30T06:25:32.005ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
18582025-04-30T06:25:32.007ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18592025-04-30T06:25:32.010ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
18602025-04-30T06:25:32.011ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6b9901d-c3cf-4ba4-8434-be8daee5d8fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18612025-04-30T06:25:32.011ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
18622025-04-30T06:25:32.013ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9d38b34-ce03-46b7-a389-7bb00778f5d5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18632025-04-30T06:25:32.013ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
18642025-04-30T06:25:32.017ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee0bf134-494a-4079-bf4c-d498479c071a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18652025-04-30T06:25:32.017ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
18662025-04-30T06:25:32.022ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76735820-db15-4ea9-bfd5-fba51406a345", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18672025-04-30T06:25:32.022ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
18682025-04-30T06:25:32.025ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a8b7b6-5181-499e-9d2c-5a9a4220cd11", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18692025-04-30T06:25:32.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
18702025-04-30T06:25:32.028ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a604f383-ce0e-40d5-9cf0-c2401a893f6d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18712025-04-30T06:25:32.028ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
18722025-04-30T06:25:32.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77463995-1a28-4dfa-a3db-6e7968013aa5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18732025-04-30T06:25:32.030ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
18742025-04-30T06:25:32.033ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c940d1f-8389-4781-9d71-905b5957421b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18752025-04-30T06:25:32.033ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
18762025-04-30T06:25:32.035ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16679adf-0dec-4125-be24-27923495b3da", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18772025-04-30T06:25:32.035ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
18782025-04-30T06:25:32.038ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea700e84-e6ff-4d4b-885b-2ae19d1b2490", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18792025-04-30T06:25:32.038ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
18802025-04-30T06:25:32.040ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eca847c7-6218-4009-aa1e-91f85d532a1e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18812025-04-30T06:25:32.041ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
18822025-04-30T06:25:32.044ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7230777-7e89-498b-a635-0566d8e27aa3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18832025-04-30T06:25:32.044ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
18842025-04-30T06:25:32.045ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88de6a76-9c84-4ba0-8b75-0fd87f6bbb07", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18852025-04-30T06:25:32.045ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
18862025-04-30T06:25:32.047ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ec4826f-45c1-4c49-80fa-cba86338b1ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18872025-04-30T06:25:32.047ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
18882025-04-30T06:25:32.048ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38c0f456-fcd2-4f82-a3c8-21a41e7440b1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18892025-04-30T06:25:32.049ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
18902025-04-30T06:25:32.050ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df8ae0ba-d1b3-4418-80b8-5b726206f257", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18912025-04-30T06:25:32.050ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
18922025-04-30T06:25:32.058ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dfa588c8-2ce0-4aef-a5dd-2a94f3e77492", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18932025-04-30T06:25:32.058ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
18942025-04-30T06:25:32.065ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db22f97b-1bc8-4883-93a5-306a9c99a955", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18952025-04-30T06:25:32.065ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
18962025-04-30T06:25:32.075ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "250c305e-0658-4605-93fe-02ee626b9ba6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18972025-04-30T06:25:32.076ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
18982025-04-30T06:25:32.078ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c71b141-8f4d-4925-8c30-db494c338896", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
18992025-04-30T06:25:32.078ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
19002025-04-30T06:25:32.081ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3beb8334-8918-4ba1-ad7e-e250a70b7a01", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19012025-04-30T06:25:32.081ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
19022025-04-30T06:25:32.082ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa637f1f-f14b-47d9-ac40-6319f2e3cd1b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19032025-04-30T06:25:32.082ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
19042025-04-30T06:25:32.083ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea0a3153-308c-4018-9aeb-84c944f71ec8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19052025-04-30T06:25:32.083ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
19062025-04-30T06:25:32.085ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a5f1623-d1a0-41c5-90c0-118ecf0b59e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19072025-04-30T06:25:32.085ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
19082025-04-30T06:25:32.086ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "114ff363-1afe-4bf4-9b48-8341d939c6dc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19092025-04-30T06:25:32.086ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
19102025-04-30T06:25:32.086ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
19112025-04-30T06:25:32.088ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f989336-d934-41e6-af4a-a14e408e3e75", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19122025-04-30T06:25:32.088ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
19132025-04-30T06:25:32.090ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "becc4cd0-6c1e-46ac-9355-6b46ee8e2490", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19142025-04-30T06:25:32.090ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
19152025-04-30T06:25:32.091ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb049924-d6f1-46d9-86c0-764784f27d33", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19162025-04-30T06:25:32.092ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
19172025-04-30T06:25:32.094ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46fa045c-a971-4220-8f1f-27cc312a9838", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19182025-04-30T06:25:32.094ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
19192025-04-30T06:25:32.096ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc28c8c6-f7c9-4538-8271-510131a18c71", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19202025-04-30T06:25:32.096ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
19212025-04-30T06:25:32.096ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19222025-04-30T06:25:32.096ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19232025-04-30T06:25:32.096ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19242025-04-30T06:25:32.104ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19252025-04-30T06:25:32.109ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19262025-04-30T06:25:32.113ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
19272025-04-30T06:25:32.113ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
19282025-04-30T06:25:32.113ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19292025-04-30T06:25:32.113ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19302025-04-30T06:25:32.113ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19312025-04-30T06:25:32.113ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19322025-04-30T06:25:32.122ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19332025-04-30T06:25:32.125ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
19342025-04-30T06:25:32.125ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19352025-04-30T06:25:32.125ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19362025-04-30T06:25:32.126ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19372025-04-30T06:25:32.128ZINFOtest_replacement_sanity (InternalApi): inserted blueprint blueprint_id = c54edae5-c899-44c6-bab8-1c9168263997
19382025-04-30T06:25:32.129ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
19392025-04-30T06:25:32.129ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19402025-04-30T06:25:32.129ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19412025-04-30T06:25:32.129ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
19422025-04-30T06:25:32.133ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19432025-04-30T06:25:32.184ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
19442025-04-30T06:25:32.185ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
19452025-04-30T06:25:32.186ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:61043 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
19462025-04-30T06:25:32.190ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19472025-04-30T06:25:32.194ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
19482025-04-30T06:25:32.195ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:61043 req_id = 36eb9795-0f4c-40c5-9c9b-04f087f61abd sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
19492025-04-30T06:25:32.196ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1369 local_addr = [::1]:42668 method = GET remote_addr = [::1]:61043 req_id = 36eb9795-0f4c-40c5-9c9b-04f087f61abd response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
19502025-04-30T06:25:32.197ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "36eb9795-0f4c-40c5-9c9b-04f087f61abd", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19512025-04-30T06:25:32.275ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
19522025-04-30T06:25:32.276ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
19532025-04-30T06:25:32.280ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:41840 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
19542025-04-30T06:25:32.281ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:41840 req_id = 5af9795c-1946-4692-aa6c-fb182a659aaa sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
19552025-04-30T06:25:32.287ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 5639 local_addr = [::1]:47918 method = GET remote_addr = [::1]:41840 req_id = 5af9795c-1946-4692-aa6c-fb182a659aaa response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
19562025-04-30T06:25:32.288ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5af9795c-1946-4692-aa6c-fb182a659aaa", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19572025-04-30T06:25:32.352ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
19582025-04-30T06:25:32.352ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
19592025-04-30T06:25:32.353ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:32864 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
19602025-04-30T06:25:32.362ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:32864 req_id = c746e3e1-30d3-428c-9374-2fe9b03a5303 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
19612025-04-30T06:25:32.362ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 229 local_addr = [::1]:35744 method = GET remote_addr = [::1]:32864 req_id = c746e3e1-30d3-428c-9374-2fe9b03a5303 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
19622025-04-30T06:25:32.385ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c746e3e1-30d3-428c-9374-2fe9b03a5303", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19632025-04-30T06:25:32.454ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
19642025-04-30T06:25:32.454ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
19652025-04-30T06:25:32.455ZINFOtest_replacement_sanity (InternalApi): Inserted networking records for external_dns service
19662025-04-30T06:25:32.455ZINFOtest_replacement_sanity (InternalApi): No networking records needed for crucible_pantry service
19672025-04-30T06:25:32.455ZINFOtest_replacement_sanity (InternalApi): No networking records needed for clickhouse service
19682025-04-30T06:25:32.455ZINFOtest_replacement_sanity (InternalApi): No networking records needed for internal_dns service
19692025-04-30T06:25:32.455ZINFOtest_replacement_sanity (InternalApi): No networking records needed for cockroach_db service
19702025-04-30T06:25:32.455ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:44992 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
19712025-04-30T06:25:32.457ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:44992 req_id = 6b3a2ab1-3f2d-4907-951b-6e7eb2a76e51 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
19722025-04-30T06:25:32.459ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2439 local_addr = [::1]:35343 method = GET remote_addr = [::1]:44992 req_id = 6b3a2ab1-3f2d-4907-951b-6e7eb2a76e51 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
19732025-04-30T06:25:32.460ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b3a2ab1-3f2d-4907-951b-6e7eb2a76e51", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:31 GMT"} })
19742025-04-30T06:25:32.462ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
19752025-04-30T06:25:32.462ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
19762025-04-30T06:25:32.462ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
19772025-04-30T06:25:32.467ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
19782025-04-30T06:25:32.467ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19792025-04-30T06:25:32.467ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19802025-04-30T06:25:32.468ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19812025-04-30T06:25:32.468ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19822025-04-30T06:25:32.477ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19832025-04-30T06:25:32.482ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19842025-04-30T06:25:32.482ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19852025-04-30T06:25:32.482ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19862025-04-30T06:25:32.484ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19872025-04-30T06:25:32.494ZINFOtest_replacement_sanity (InternalApi): Inserted networking records for nexus service
19882025-04-30T06:25:32.494ZINFOtest_replacement_sanity (InternalApi): Inserted service networking records
19892025-04-30T06:25:32.494ZINFOtest_replacement_sanity (InternalApi): Inserted physical disks
19902025-04-30T06:25:32.494ZINFOtest_replacement_sanity (InternalApi): Inserted zpools
19912025-04-30T06:25:32.494ZINFOtest_replacement_sanity (InternalApi): Inserted datasets
19922025-04-30T06:25:32.514ZINFOtest_replacement_sanity (InternalApi): Populated DNS tables for internal DNS
19932025-04-30T06:25:32.528ZINFOtest_replacement_sanity (InternalApi): Populated DNS tables for external DNS
19942025-04-30T06:25:32.529ZTRACtest_replacement_sanity (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
19952025-04-30T06:25:32.530ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19962025-04-30T06:25:32.530ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19972025-04-30T06:25:32.530ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
19982025-04-30T06:25:32.530ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19992025-04-30T06:25:32.535ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20002025-04-30T06:25:32.538ZDEBGtest_replacement_sanity (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
20012025-04-30T06:25:32.538ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
20022025-04-30T06:25:32.538ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20032025-04-30T06:25:32.538ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20042025-04-30T06:25:32.538ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20052025-04-30T06:25:32.539ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20062025-04-30T06:25:32.551ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20072025-04-30T06:25:32.554ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
20082025-04-30T06:25:32.589ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
20092025-04-30T06:25:32.589ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20102025-04-30T06:25:32.589ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20112025-04-30T06:25:32.589ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20122025-04-30T06:25:32.590ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20132025-04-30T06:25:32.593ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20142025-04-30T06:25:32.601ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
20152025-04-30T06:25:32.601ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
20162025-04-30T06:25:32.601ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20172025-04-30T06:25:32.601ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20182025-04-30T06:25:32.601ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20192025-04-30T06:25:32.602ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20202025-04-30T06:25:32.605ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20212025-04-30T06:25:32.612ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
20222025-04-30T06:25:32.615ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
20232025-04-30T06:25:32.615ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20242025-04-30T06:25:32.616ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20252025-04-30T06:25:32.616ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20262025-04-30T06:25:32.617ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20272025-04-30T06:25:32.619ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20282025-04-30T06:25:32.626ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
20292025-04-30T06:25:32.684ZINFOtest_replacement_sanity (InternalApi): Created recovery silo
20302025-04-30T06:25:32.686ZINFOtest_replacement_sanity (InternalApi): Created recovery user
20312025-04-30T06:25:32.721ZINFOtest_replacement_sanity (InternalApi): Created recovery user's password
20322025-04-30T06:25:32.721ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 2f316342-2ac1-4771-8de2-ab5b69dc236f, lookup_type: ById(2f316342-2ac1-4771-8de2-ab5b69dc236f) }
20332025-04-30T06:25:32.722ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 2f316342-2ac1-4771-8de2-ab5b69dc236f resource_type = Silo
20342025-04-30T06:25:32.722ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20352025-04-30T06:25:32.722ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20362025-04-30T06:25:32.723ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20372025-04-30T06:25:32.728ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20382025-04-30T06:25:32.728ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20392025-04-30T06:25:32.728ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20402025-04-30T06:25:32.729ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20412025-04-30T06:25:32.732ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20422025-04-30T06:25:32.744ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 2f316342-2ac1-4771-8de2-ab5b69dc236f, lookup_type: ById(2f316342-2ac1-4771-8de2-ab5b69dc236f) } result = Ok(())
20432025-04-30T06:25:32.745ZDEBGtest_replacement_sanity (InternalApi): Generated role assignment queries
20442025-04-30T06:25:32.755ZINFOtest_replacement_sanity (InternalApi): Granted Silo privileges
20452025-04-30T06:25:32.755ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
20462025-04-30T06:25:32.755ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20472025-04-30T06:25:32.755ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20482025-04-30T06:25:32.755ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20492025-04-30T06:25:32.757ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20502025-04-30T06:25:32.778ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20512025-04-30T06:25:32.784ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
20522025-04-30T06:25:32.803ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20532025-04-30T06:25:32.805ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20542025-04-30T06:25:32.813ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20552025-04-30T06:25:32.824ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20562025-04-30T06:25:32.824ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20572025-04-30T06:25:32.825ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20582025-04-30T06:25:32.829ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20592025-04-30T06:25:32.829ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20602025-04-30T06:25:32.830ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20612025-04-30T06:25:32.834ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20622025-04-30T06:25:32.834ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20632025-04-30T06:25:32.834ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20642025-04-30T06:25:32.834ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20652025-04-30T06:25:32.837ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20662025-04-30T06:25:32.841ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20672025-04-30T06:25:32.841ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20682025-04-30T06:25:32.841ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20692025-04-30T06:25:32.842ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20702025-04-30T06:25:32.844ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20712025-04-30T06:25:32.844ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20722025-04-30T06:25:32.844ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20732025-04-30T06:25:32.844ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20742025-04-30T06:25:32.850ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20752025-04-30T06:25:32.862ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
20762025-04-30T06:25:32.867ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
20772025-04-30T06:25:32.867ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20782025-04-30T06:25:32.867ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20792025-04-30T06:25:32.867ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20802025-04-30T06:25:32.867ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20812025-04-30T06:25:32.868ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20822025-04-30T06:25:32.874ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20832025-04-30T06:25:32.874ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20842025-04-30T06:25:32.874ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20852025-04-30T06:25:32.875ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20862025-04-30T06:25:32.882ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20872025-04-30T06:25:32.882ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20882025-04-30T06:25:32.882ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20892025-04-30T06:25:32.882ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20902025-04-30T06:25:32.897ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20912025-04-30T06:25:32.905ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
20922025-04-30T06:25:32.906ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20932025-04-30T06:25:32.907ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20942025-04-30T06:25:32.908ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20952025-04-30T06:25:32.915ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20962025-04-30T06:25:32.925ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
20972025-04-30T06:25:32.926ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20982025-04-30T06:25:32.937ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20992025-04-30T06:25:32.937ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21002025-04-30T06:25:32.938ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21012025-04-30T06:25:32.947ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21022025-04-30T06:25:32.947ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21032025-04-30T06:25:32.948ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21042025-04-30T06:25:32.957ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21052025-04-30T06:25:32.957ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21062025-04-30T06:25:32.957ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21072025-04-30T06:25:32.957ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21082025-04-30T06:25:32.958ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21092025-04-30T06:25:32.963ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21102025-04-30T06:25:32.963ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21112025-04-30T06:25:32.963ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21122025-04-30T06:25:32.963ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21132025-04-30T06:25:32.967ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21142025-04-30T06:25:32.967ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21152025-04-30T06:25:32.967ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21162025-04-30T06:25:32.967ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21172025-04-30T06:25:32.969ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21182025-04-30T06:25:32.977ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21192025-04-30T06:25:32.978ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21202025-04-30T06:25:32.978ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21212025-04-30T06:25:32.978ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21222025-04-30T06:25:32.978ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21232025-04-30T06:25:32.993ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21242025-04-30T06:25:32.993ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21252025-04-30T06:25:32.994ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21262025-04-30T06:25:32.999ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21272025-04-30T06:25:32.999ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21282025-04-30T06:25:32.999ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21292025-04-30T06:25:33.029ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21302025-04-30T06:25:33.029ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21312025-04-30T06:25:33.029ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21322025-04-30T06:25:33.056ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21332025-04-30T06:25:33.056ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21342025-04-30T06:25:33.056ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21352025-04-30T06:25:33.057ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21362025-04-30T06:25:33.057ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21372025-04-30T06:25:33.082ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21382025-04-30T06:25:33.082ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21392025-04-30T06:25:33.082ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21402025-04-30T06:25:33.083ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21412025-04-30T06:25:33.097ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21422025-04-30T06:25:33.098ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21432025-04-30T06:25:33.098ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21442025-04-30T06:25:33.099ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21452025-04-30T06:25:33.107ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21462025-04-30T06:25:33.124ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
21472025-04-30T06:25:33.124ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
21482025-04-30T06:25:33.124ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
21492025-04-30T06:25:33.124ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21502025-04-30T06:25:33.124ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21512025-04-30T06:25:33.124ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21522025-04-30T06:25:33.125ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21532025-04-30T06:25:33.131ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 99 7c 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 .|..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:59038 req_id = 7d79fe79-a357-464a-8b09-7415aee786e9
21542025-04-30T06:25:33.131ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39292,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59038 req_id = 7d79fe79-a357-464a-8b09-7415aee786e9
21552025-04-30T06:25:33.131ZDEBGtest_replacement_sanity (store): query key key = _crucible-pantry._tcp
21562025-04-30T06:25:33.131ZDEBGtest_replacement_sanity (store): query key key = 774760d4-5767-4f71-9092-86a22ff16c67.host
21572025-04-30T06:25:33.131ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59038 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 59920, target: Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal") })) }] req_id = 7d79fe79-a357-464a-8b09-7415aee786e9
21582025-04-30T06:25:33.133ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21592025-04-30T06:25:33.133ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21602025-04-30T06:25:33.133ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21612025-04-30T06:25:33.134ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21622025-04-30T06:25:33.135ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 0c 46 01 00 00 01 00 00 00 00 00 01 24 37 37 34 .F..........$774\n0010: 37 36 30 64 34 2d 35 37 36 37 2d 34 66 37 31 2d 760d4-5767-4f71-\n0020: 39 30 39 32 2d 38 36 61 32 32 66 66 31 36 63 36 9092-86a22ff16c6\n0030: 37 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 7.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:56163 req_id = 86aea6b8-4577-405a-8404-edf9a476800b
21632025-04-30T06:25:33.135ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3142,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 55,\n 55,\n 52,\n 55,\n 54,\n 48,\n 100,\n 52,\n 45,\n 53,\n 55,\n 54,\n 55,\n 45,\n 52,\n 102,\n 55,\n 49,\n 45,\n 57,\n 48,\n 57,\n 50,\n 45,\n 56,\n 54,\n 97,\n 50,\n 50,\n 102,\n 102,\n 49,\n 54,\n 99,\n 54,\n 55,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56163 req_id = 86aea6b8-4577-405a-8404-edf9a476800b
21642025-04-30T06:25:33.135ZDEBGtest_replacement_sanity (store): query key key = 774760d4-5767-4f71-9092-86a22ff16c67.host
21652025-04-30T06:25:33.135ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56163 query = LowerQuery { name: LowerName(Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal.")), original: Query { name: Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("774760d4-5767-4f71-9092-86a22ff16c67.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 86aea6b8-4577-405a-8404-edf9a476800b
21662025-04-30T06:25:33.244ZINFOtest_replacement_sanity (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 8e0b9a02-fe6f-4a9f-9818-5453e787b6db
21672025-04-30T06:25:33.244ZDEBGtest_replacement_sanity (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 8e0b9a02-fe6f-4a9f-9818-5453e787b6db time_started = 2025-04-30 06:25:28.540256 UTC
21682025-04-30T06:25:33.244ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 5.066329691s iteration = 5 status = {"collection_id":"8e0b9a02-fe6f-4a9f-9818-5453e787b6db","time_done":"2025-04-30 06:25:32.462828 UTC","time_started":"2025-04-30 06:25:28.540256 UTC"}
21692025-04-30T06:25:33.245ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
21702025-04-30T06:25:33.245ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 123.607\u{b5}s iteration = 2 status = {"error":"task disabled"}
21712025-04-30T06:25:33.245ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
21722025-04-30T06:25:33.245ZWARNtest_replacement_sanity (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
21732025-04-30T06:25:33.266ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 20.721987ms iteration = 2 status = {"error":"no blueprint"}
21742025-04-30T06:25:33.267ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:37426 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21752025-04-30T06:25:33.291ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21762025-04-30T06:25:33.292ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21772025-04-30T06:25:33.292ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21782025-04-30T06:25:33.292ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21792025-04-30T06:25:33.293ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:37426 req_id = b6b68783-1c95-4873-8806-d47e4f6b3431 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21802025-04-30T06:25:33.294ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 661 local_addr = [::1]:59920 method = GET remote_addr = [::1]:37426 req_id = b6b68783-1c95-4873-8806-d47e4f6b3431 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21812025-04-30T06:25:33.834ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:59103 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21822025-04-30T06:25:33.835ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:39193 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21832025-04-30T06:25:33.835ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:55804 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21842025-04-30T06:25:33.835ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:49494 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21852025-04-30T06:25:33.835ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:34439 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21862025-04-30T06:25:33.835ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:36500 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21872025-04-30T06:25:33.835ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:50019 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
21882025-04-30T06:25:33.836ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21892025-04-30T06:25:33.844ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
21902025-04-30T06:25:33.844ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21912025-04-30T06:25:33.844ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
21922025-04-30T06:25:33.844ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21932025-04-30T06:25:33.846ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:59103 req_id = 72bae992-a44c-4a57-a456-6a5fa803bf44 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21942025-04-30T06:25:33.846ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:39193 req_id = 2d4669d4-988d-428b-8750-8ccc9b79c907 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21952025-04-30T06:25:33.847ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:55804 req_id = a40e487e-8542-4523-9086-2b30418b34e1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21962025-04-30T06:25:33.847ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:49494 req_id = 7622297b-7816-4328-a667-acbeaeac1c8d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21972025-04-30T06:25:33.847ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:34439 req_id = 1da9b892-e8f9-4861-89f8-d821c48da65d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21982025-04-30T06:25:33.847ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:36500 req_id = a460cacc-2d29-4af1-904f-30a828934d3f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
21992025-04-30T06:25:33.847ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:50019 req_id = 6d78d72a-2114-4308-8ec5-5a8c89071ab2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22002025-04-30T06:25:33.853ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 7557 local_addr = [::1]:59920 method = GET remote_addr = [::1]:59103 req_id = 72bae992-a44c-4a57-a456-6a5fa803bf44 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22012025-04-30T06:25:33.853ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 7207 local_addr = [::1]:59920 method = GET remote_addr = [::1]:39193 req_id = 2d4669d4-988d-428b-8750-8ccc9b79c907 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22022025-04-30T06:25:33.854ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 6970 local_addr = [::1]:59920 method = GET remote_addr = [::1]:55804 req_id = a40e487e-8542-4523-9086-2b30418b34e1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22032025-04-30T06:25:33.854ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 6886 local_addr = [::1]:59920 method = GET remote_addr = [::1]:49494 req_id = 7622297b-7816-4328-a667-acbeaeac1c8d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22042025-04-30T06:25:33.854ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 6806 local_addr = [::1]:59920 method = GET remote_addr = [::1]:34439 req_id = 1da9b892-e8f9-4861-89f8-d821c48da65d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22052025-04-30T06:25:33.854ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 6725 local_addr = [::1]:59920 method = GET remote_addr = [::1]:36500 req_id = a460cacc-2d29-4af1-904f-30a828934d3f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22062025-04-30T06:25:33.854ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 6650 local_addr = [::1]:59920 method = GET remote_addr = [::1]:50019 req_id = 6d78d72a-2114-4308-8ec5-5a8c89071ab2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
22072025-04-30T06:25:33.862ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22082025-04-30T06:25:33.862ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22092025-04-30T06:25:33.862ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22102025-04-30T06:25:33.876ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22112025-04-30T06:25:33.876ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22122025-04-30T06:25:33.877ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22132025-04-30T06:25:33.888ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22142025-04-30T06:25:33.888ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22152025-04-30T06:25:33.889ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22162025-04-30T06:25:33.893ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22172025-04-30T06:25:33.893ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22182025-04-30T06:25:33.894ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22192025-04-30T06:25:33.906ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22202025-04-30T06:25:33.906ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22212025-04-30T06:25:33.906ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22222025-04-30T06:25:33.906ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22232025-04-30T06:25:33.907ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22242025-04-30T06:25:33.913ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22252025-04-30T06:25:33.913ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22262025-04-30T06:25:33.913ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22272025-04-30T06:25:33.914ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22282025-04-30T06:25:33.928ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22292025-04-30T06:25:33.928ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22302025-04-30T06:25:33.928ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22312025-04-30T06:25:33.929ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22322025-04-30T06:25:33.931ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22332025-04-30T06:25:33.954ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22342025-04-30T06:25:33.954ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22352025-04-30T06:25:33.954ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22362025-04-30T06:25:33.954ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22372025-04-30T06:25:33.955ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22382025-04-30T06:25:33.955ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22392025-04-30T06:25:33.955ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22402025-04-30T06:25:33.962ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22412025-04-30T06:25:33.962ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22422025-04-30T06:25:33.962ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22432025-04-30T06:25:33.963ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22442025-04-30T06:25:33.965ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22452025-04-30T06:25:33.965ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22462025-04-30T06:25:33.965ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22472025-04-30T06:25:33.966ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22482025-04-30T06:25:33.981ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22492025-04-30T06:25:33.988ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22502025-04-30T06:25:33.988ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22512025-04-30T06:25:33.989ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22522025-04-30T06:25:33.989ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22532025-04-30T06:25:34.003ZDEBGtest_replacement_sanity (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 75d58370-2da0-4751-8dd3-8f5d02e520b7, kind: Service { id: c31da0ae-930d-4dd9-a897-159d5a6cd420 }, name: Name("nexus-c31da0ae-930d-4dd9-a897-159d5a6cd420"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 6ee0a63c-7d7b-4d03-a405-8d41adf7f5a5, kind: Service { id: 3ae50786-e3ad-43a3-866e-b9ad16af275d }, name: Name("external-dns-3ae50786-e3ad-43a3-866e-b9ad16af275d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
22542025-04-30T06:25:34.003ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
22552025-04-30T06:25:34.003ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22562025-04-30T06:25:34.003ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22572025-04-30T06:25:34.003ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22582025-04-30T06:25:34.004ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22592025-04-30T06:25:34.008ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22602025-04-30T06:25:34.011ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
22612025-04-30T06:25:34.011ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22622025-04-30T06:25:34.011ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
22632025-04-30T06:25:34.012ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22642025-04-30T06:25:34.020ZTRACtest_replacement_sanity (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:25:32.784618Z, allowed_ips: None }
22652025-04-30T06:25:34.020ZDEBGtest_replacement_sanity (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
22662025-04-30T06:25:34.021ZDEBGtest_replacement_sanity (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 6ee0a63c-7d7b-4d03-a405-8d41adf7f5a5, kind: Service { id: 3ae50786-e3ad-43a3-866e-b9ad16af275d }, name: Name("external-dns-3ae50786-e3ad-43a3-866e-b9ad16af275d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 75d58370-2da0-4751-8dd3-8f5d02e520b7, kind: Service { id: c31da0ae-930d-4dd9-a897-159d5a6cd420 }, name: Name("nexus-c31da0ae-930d-4dd9-a897-159d5a6cd420"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
22672025-04-30T06:25:34.021ZDEBGtest_replacement_sanity (nexus): resolved 2 rules for sleds
22682025-04-30T06:25:34.087ZDEBGtest_replacement_sanity (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:25:25.395837Z, time_modified: 2025-04-30T06:25:25.395837Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(35343), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(46651) }]
22692025-04-30T06:25:34.087ZDEBGtest_replacement_sanity (nexus): sending firewall rules to sled agents
22702025-04-30T06:25:34.088ZTRACtest_replacement_sanity (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
22712025-04-30T06:25:34.088ZDEBGtest_replacement_sanity (InstanceAllocator): roles roles = RoleSet { roles: {} }
22722025-04-30T06:25:34.088ZDEBGtest_replacement_sanity (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
22732025-04-30T06:25:34.093ZTRACtest_replacement_sanity (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
22742025-04-30T06:25:34.093ZTRACtest_replacement_sanity (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22752025-04-30T06:25:34.094ZTRACtest_replacement_sanity (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
22762025-04-30T06:25:34.094ZDEBGtest_replacement_sanity (InstanceAllocator): roles roles = RoleSet { roles: {} }
22772025-04-30T06:25:34.095ZDEBGtest_replacement_sanity (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
22782025-04-30T06:25:34.097ZDEBGtest_replacement_sanity (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
22792025-04-30T06:25:34.102ZDEBGtest_replacement_sanity (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
22802025-04-30T06:25:34.159ZDEBGtest_replacement_sanity (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:35343/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22812025-04-30T06:25:34.160ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:62770 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
22822025-04-30T06:25:34.163ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:62770 req_id = 06dc200c-6635-44db-8574-01450df27f36 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22832025-04-30T06:25:34.165ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1961 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:62770 req_id = 06dc200c-6635-44db-8574-01450df27f36 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
22842025-04-30T06:25:34.169ZDEBGtest_replacement_sanity (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:35343/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "06dc200c-6635-44db-8574-01450df27f36", "date": "Wed, 30 Apr 2025 06:25:33 GMT"} })
22852025-04-30T06:25:34.171ZINFOtest_replacement_sanity (nexus): updated firewall rules on 1 sleds
22862025-04-30T06:25:34.171ZTRACtest_replacement_sanity (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
22872025-04-30T06:25:34.171ZDEBGtest_replacement_sanity (ServiceBalancer): roles roles = RoleSet { roles: {} }
22882025-04-30T06:25:34.172ZDEBGtest_replacement_sanity (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
22892025-04-30T06:25:34.172ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
22902025-04-30T06:25:34.172ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
22912025-04-30T06:25:34.172ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
22922025-04-30T06:25:34.173ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
22932025-04-30T06:25:34.173ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
22942025-04-30T06:25:34.173ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
22952025-04-30T06:25:34.173ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
22962025-04-30T06:25:34.173ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
22972025-04-30T06:25:34.173ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
22982025-04-30T06:25:34.173ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
22992025-04-30T06:25:34.173ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23002025-04-30T06:25:34.174ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23012025-04-30T06:25:34.174ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
23022025-04-30T06:25:34.174ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
23032025-04-30T06:25:34.174ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
23042025-04-30T06:25:34.174ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
23052025-04-30T06:25:34.174ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23062025-04-30T06:25:34.174ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
23072025-04-30T06:25:34.174ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23082025-04-30T06:25:34.174ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23092025-04-30T06:25:34.175ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
23102025-04-30T06:25:34.175ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
23112025-04-30T06:25:34.175ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23122025-04-30T06:25:34.175ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23132025-04-30T06:25:34.175ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23142025-04-30T06:25:34.175ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23152025-04-30T06:25:34.190ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 8e da 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:63930 req_id = 007551b9-acbc-4f59-a883-dfd44c1788be
23162025-04-30T06:25:34.190ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36570,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63930 req_id = 007551b9-acbc-4f59-a883-dfd44c1788be
23172025-04-30T06:25:34.190ZDEBGtest_replacement_sanity (store): query key key = _nameservice._tcp
23182025-04-30T06:25:34.190ZDEBGtest_replacement_sanity (store): query key key = a4736702-dd27-45ae-aa41-29a35dcc68a1.host
23192025-04-30T06:25:34.191ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63930 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 34917, target: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal") })) }] req_id = 007551b9-acbc-4f59-a883-dfd44c1788be
23202025-04-30T06:25:34.191ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 42 cf 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 B............_ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58048 req_id = 26b708c5-6bfa-4d3f-b948-f7bb89e2c4ba
23212025-04-30T06:25:34.191ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17103,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58048 req_id = 26b708c5-6bfa-4d3f-b948-f7bb89e2c4ba
23222025-04-30T06:25:34.191ZDEBGtest_replacement_sanity (store): query key key = _external-dns._tcp
23232025-04-30T06:25:34.191ZDEBGtest_replacement_sanity (store): query key key = 3ae50786-e3ad-43a3-866e-b9ad16af275d.host
23242025-04-30T06:25:34.191ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58048 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35427, target: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal") })) }] req_id = 26b708c5-6bfa-4d3f-b948-f7bb89e2c4ba
23252025-04-30T06:25:34.201ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
23262025-04-30T06:25:34.201ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23272025-04-30T06:25:34.201ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
23282025-04-30T06:25:34.201ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23292025-04-30T06:25:34.202ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23302025-04-30T06:25:34.202ZTRACtest_replacement_sanity (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
23312025-04-30T06:25:34.202ZTRACtest_replacement_sanity (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23322025-04-30T06:25:34.202ZTRACtest_replacement_sanity (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
23332025-04-30T06:25:34.202ZDEBGtest_replacement_sanity (ServiceBalancer): roles roles = RoleSet { roles: {} }
23342025-04-30T06:25:34.203ZDEBGtest_replacement_sanity (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
23352025-04-30T06:25:34.204ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 34917, target: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3199, tv_nsec: 3662667 } })
23362025-04-30T06:25:34.205ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
23372025-04-30T06:25:34.206ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 35427, target: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3199, tv_nsec: 4004047 } })
23382025-04-30T06:25:34.206ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
23392025-04-30T06:25:34.209ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
23402025-04-30T06:25:34.209ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23412025-04-30T06:25:34.209ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
23422025-04-30T06:25:34.209ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
23432025-04-30T06:25:34.210ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
23442025-04-30T06:25:34.230ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 75 7e 01 00 00 01 00 00 00 00 00 01 24 61 34 37 u~..........$a47\n0010: 33 36 37 30 32 2d 64 64 32 37 2d 34 35 61 65 2d 36702-dd27-45ae-\n0020: 61 61 34 31 2d 32 39 61 33 35 64 63 63 36 38 61 aa41-29a35dcc68a\n0030: 31 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 1.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:62124 req_id = cc98711c-19ab-4715-bdd5-2d57f749e66b
23452025-04-30T06:25:34.230ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30078,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 97,\n 52,\n 55,\n 51,\n 54,\n 55,\n 48,\n 50,\n 45,\n 100,\n 100,\n 50,\n 55,\n 45,\n 52,\n 53,\n 97,\n 101,\n 45,\n 97,\n 97,\n 52,\n 49,\n 45,\n 50,\n 57,\n 97,\n 51,\n 53,\n 100,\n 99,\n 99,\n 54,\n 56,\n 97,\n 49,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62124 req_id = cc98711c-19ab-4715-bdd5-2d57f749e66b
23462025-04-30T06:25:34.230ZDEBGtest_replacement_sanity (store): query key key = a4736702-dd27-45ae-aa41-29a35dcc68a1.host
23472025-04-30T06:25:34.230ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62124 query = LowerQuery { name: LowerName(Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal.")), original: Query { name: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("a4736702-dd27-45ae-aa41-29a35dcc68a1.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = cc98711c-19ab-4715-bdd5-2d57f749e66b
23482025-04-30T06:25:34.230ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 4e a7 01 00 00 01 00 00 00 00 00 01 24 33 61 65 N...........$3ae\n0010: 35 30 37 38 36 2d 65 33 61 64 2d 34 33 61 33 2d 50786-e3ad-43a3-\n0020: 38 36 36 65 2d 62 39 61 64 31 36 61 66 32 37 35 866e-b9ad16af275\n0030: 64 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 d.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:58565 req_id = 8a6858db-2575-4647-bd15-69a0e54da84c
23492025-04-30T06:25:34.230ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20135,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 51,\n 97,\n 101,\n 53,\n 48,\n 55,\n 56,\n 54,\n 45,\n 101,\n 51,\n 97,\n 100,\n 45,\n 52,\n 51,\n 97,\n 51,\n 45,\n 56,\n 54,\n 54,\n 101,\n 45,\n 98,\n 57,\n 97,\n 100,\n 49,\n 54,\n 97,\n 102,\n 50,\n 55,\n 53,\n 100,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58565 req_id = 8a6858db-2575-4647-bd15-69a0e54da84c
23502025-04-30T06:25:34.230ZDEBGtest_replacement_sanity (store): query key key = 3ae50786-e3ad-43a3-866e-b9ad16af275d.host
23512025-04-30T06:25:34.230ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58565 query = LowerQuery { name: LowerName(Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal.")), original: Query { name: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("3ae50786-e3ad-43a3-866e-b9ad16af275d.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8a6858db-2575-4647-bd15-69a0e54da84c
23522025-04-30T06:25:34.231ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23532025-04-30T06:25:34.252ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
23542025-04-30T06:25:34.252ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
23552025-04-30T06:25:34.252ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23562025-04-30T06:25:34.253ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23572025-04-30T06:25:34.253ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23582025-04-30T06:25:34.257ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
23592025-04-30T06:25:34.257ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23602025-04-30T06:25:34.257ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23612025-04-30T06:25:34.257ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23622025-04-30T06:25:34.258ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23632025-04-30T06:25:34.264ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23642025-04-30T06:25:34.266ZDEBGtest_replacement_sanity (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
23652025-04-30T06:25:34.267ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
23662025-04-30T06:25:34.267ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
23672025-04-30T06:25:34.267ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
23682025-04-30T06:25:34.267ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23692025-04-30T06:25:34.267ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
23702025-04-30T06:25:34.267ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
23712025-04-30T06:25:34.268ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
23722025-04-30T06:25:34.269ZINFOtest_replacement_sanity (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:34917] } background_task = dns_servers_internal dns_group = internal
23732025-04-30T06:25:34.269ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 96.043961ms iteration = 2 status = {"addresses":["[::1]:34917"]}
23742025-04-30T06:25:34.269ZINFOtest_replacement_sanity (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:35427] } background_task = dns_servers_external dns_group = external
23752025-04-30T06:25:34.269ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 95.603123ms iteration = 2 status = {"addresses":["[::1]:35427"]}
23762025-04-30T06:25:34.271ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
23772025-04-30T06:25:34.271ZWARNtest_replacement_sanity (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
23782025-04-30T06:25:34.271ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 210.129\u{b5}s iteration = 2 status = {"error":"no config"}
23792025-04-30T06:25:34.271ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
23802025-04-30T06:25:34.271ZWARNtest_replacement_sanity (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
23812025-04-30T06:25:34.271ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 161.381\u{b5}s iteration = 2 status = {"error":"no config"}
23822025-04-30T06:25:34.275ZDEBGtest_replacement_sanity (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23832025-04-30T06:25:34.304ZDEBGtest_replacement_sanity (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
23842025-04-30T06:25:34.304ZINFOtest_replacement_sanity (nexus): Rack initialized
23852025-04-30T06:25:34.304ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23862025-04-30T06:25:34.304ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
23872025-04-30T06:25:34.305ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23882025-04-30T06:25:34.307ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23892025-04-30T06:25:34.322ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
23902025-04-30T06:25:34.323ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
23912025-04-30T06:25:34.323ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23922025-04-30T06:25:34.323ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
23932025-04-30T06:25:34.323ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
23942025-04-30T06:25:34.324ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
23952025-04-30T06:25:34.327ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
23962025-04-30T06:25:34.327ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
23972025-04-30T06:25:34.328ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23982025-04-30T06:25:34.378ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
23992025-04-30T06:25:34.378ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
24002025-04-30T06:25:34.378ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24012025-04-30T06:25:34.379ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24022025-04-30T06:25:34.384ZDEBGtest_replacement_sanity (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
24032025-04-30T06:25:34.384ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
24042025-04-30T06:25:34.384ZDEBGtest_replacement_sanity (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
24052025-04-30T06:25:34.384ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
24062025-04-30T06:25:34.384ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24072025-04-30T06:25:34.384ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
24082025-04-30T06:25:34.384ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24092025-04-30T06:25:34.385ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24102025-04-30T06:25:34.394ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24112025-04-30T06:25:34.400ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
24122025-04-30T06:25:34.400ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
24132025-04-30T06:25:34.400ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24142025-04-30T06:25:34.400ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24152025-04-30T06:25:34.403ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24162025-04-30T06:25:34.409ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
24172025-04-30T06:25:34.409ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
24182025-04-30T06:25:34.409ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24192025-04-30T06:25:34.425ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24202025-04-30T06:25:34.427ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
24212025-04-30T06:25:34.427ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
24222025-04-30T06:25:34.428ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
24232025-04-30T06:25:34.433ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6b 77 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 kw..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53473 req_id = 3ef1fefc-9735-48fb-8b72-734cee05a58a
24242025-04-30T06:25:34.433ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27511,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53473 req_id = 3ef1fefc-9735-48fb-8b72-734cee05a58a
24252025-04-30T06:25:34.434ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
24262025-04-30T06:25:34.434ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
24272025-04-30T06:25:34.434ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
24282025-04-30T06:25:34.434ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53473 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3ef1fefc-9735-48fb-8b72-734cee05a58a
24292025-04-30T06:25:34.442ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
24302025-04-30T06:25:34.442ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24312025-04-30T06:25:34.442ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
24322025-04-30T06:25:34.442ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24332025-04-30T06:25:34.442ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24342025-04-30T06:25:34.442ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3199, tv_nsec: 237309242 } })
24352025-04-30T06:25:34.442ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
24362025-04-30T06:25:34.442ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
24372025-04-30T06:25:34.442ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
24382025-04-30T06:25:34.442ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
24392025-04-30T06:25:34.442ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
24402025-04-30T06:25:34.450ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24412025-04-30T06:25:34.457ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
24422025-04-30T06:25:34.461ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
24432025-04-30T06:25:34.462ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
24442025-04-30T06:25:34.462ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
24452025-04-30T06:25:34.463ZDEBGtest_replacement_sanity (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
24462025-04-30T06:25:34.463ZDEBGtest_replacement_sanity (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = 95391d84-9647-40a7-a2a2-1c1f1a25a5c0 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
24472025-04-30T06:25:34.463ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
24482025-04-30T06:25:34.463ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24492025-04-30T06:25:34.463ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
24502025-04-30T06:25:34.463ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24512025-04-30T06:25:34.464ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24522025-04-30T06:25:34.466ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b2 4e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .N..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53737 req_id = f593fe09-df8c-416c-b062-3d46c331a1d0
24532025-04-30T06:25:34.466ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45646,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53737 req_id = f593fe09-df8c-416c-b062-3d46c331a1d0
24542025-04-30T06:25:34.467ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
24552025-04-30T06:25:34.467ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53737 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f593fe09-df8c-416c-b062-3d46c331a1d0
24562025-04-30T06:25:34.467ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d3 60 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .`..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61130 req_id = 199f83e9-1318-43d6-953c-2558acb191c6
24572025-04-30T06:25:34.468ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54112,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61130 req_id = 199f83e9-1318-43d6-953c-2558acb191c6
24582025-04-30T06:25:34.468ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
24592025-04-30T06:25:34.468ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61130 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 199f83e9-1318-43d6-953c-2558acb191c6
24602025-04-30T06:25:34.468ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
24612025-04-30T06:25:34.469ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
24622025-04-30T06:25:34.469ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
24632025-04-30T06:25:34.469ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
24642025-04-30T06:25:34.469ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
24652025-04-30T06:25:34.680ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
24662025-04-30T06:25:34.681ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24672025-04-30T06:25:34.690ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
24682025-04-30T06:25:34.709ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
24692025-04-30T06:25:34.709ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
24702025-04-30T06:25:34.710ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
24712025-04-30T06:25:34.711ZDEBGtest_replacement_sanity (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
24722025-04-30T06:25:34.711ZDEBGtest_replacement_sanity (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 82724bc3-c920-4fde-a779-1ccedc0b0889 dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
24732025-04-30T06:25:34.711ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
24742025-04-30T06:25:34.711ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
24752025-04-30T06:25:34.711ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
24762025-04-30T06:25:34.711ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24772025-04-30T06:25:34.712ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24782025-04-30T06:25:34.716ZINFOtest_replacement_sanity (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:25:23.430667Z, time_modified: 2025-04-30T06:25:23.430667Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 2f316342-2ac1-4771-8de2-ab5b69dc236f, db_silo: Silo { identity: SiloIdentity { id: 2f316342-2ac1-4771-8de2-ab5b69dc236f, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:25:32.529629Z, time_modified: 2025-04-30T06:25:32.529629Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 2f316342-2ac1-4771-8de2-ab5b69dc236f, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 2f316342-2ac1-4771-8de2-ab5b69dc236f, db_silo: Silo { identity: SiloIdentity { id: 2f316342-2ac1-4771-8de2-ab5b69dc236f, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:25:32.529629Z, time_modified: 2025-04-30T06:25:32.529629Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
24792025-04-30T06:25:34.716ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 541.377099ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"2f316342-2ac1-4771-8de2-ab5b69dc236f","tls_certs":[]}},"default_endpoint":{"silo_id":"2f316342-2ac1-4771-8de2-ab5b69dc236f","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 2f316342-2ac1-4771-8de2-ab5b69dc236f with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
24802025-04-30T06:25:34.719ZDEBGtest_replacement_sanity (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = 95391d84-9647-40a7-a2a2-1c1f1a25a5c0 dns_zone_name = control-plane.oxide.internal found_so_far = 23 version = 1
24812025-04-30T06:25:34.720ZDEBGtest_replacement_sanity (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
24822025-04-30T06:25:34.720ZINFOtest_replacement_sanity (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
24832025-04-30T06:25:34.720ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 547.276423ms iteration = 2 status = {"generation":1}
24842025-04-30T06:25:34.720ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
24852025-04-30T06:25:34.833ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:34917 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:34917] } uri = http://[::1]:34917/config
24862025-04-30T06:25:34.840ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
24872025-04-30T06:25:34.861ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
24882025-04-30T06:25:34.861ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
24892025-04-30T06:25:34.861ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
24902025-04-30T06:25:34.862ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
24912025-04-30T06:25:34.863ZINFOtest_replacement_sanity (http): accepted connection kind = dns local_addr = [::1]:34917 remote_addr = [::1]:34158
24922025-04-30T06:25:34.870ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
24932025-04-30T06:25:34.870ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
24942025-04-30T06:25:34.870ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
24952025-04-30T06:25:34.871ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
24962025-04-30T06:25:34.872ZTRACtest_replacement_sanity (http): incoming request kind = dns local_addr = [::1]:34917 method = PUT remote_addr = [::1]:34158 req_id = 06ba02d9-679e-44ab-bd0b-595241cc144d uri = /config
24972025-04-30T06:25:34.872ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 43 c9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c C............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63354 req_id = f9c3682d-e0e7-43b3-b56f-b3eaf7097e82
24982025-04-30T06:25:34.872ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17353,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63354 req_id = f9c3682d-e0e7-43b3-b56f-b3eaf7097e82
24992025-04-30T06:25:34.872ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
25002025-04-30T06:25:34.872ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17353, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63354 req_id = f9c3682d-e0e7-43b3-b56f-b3eaf7097e82
25012025-04-30T06:25:34.873ZINFOtest_replacement_sanity (store): attempting generation update new_generation = 1 req_id = 06ba02d9-679e-44ab-bd0b-595241cc144d
25022025-04-30T06:25:34.873ZINFOtest_replacement_sanity (store): updated generation new_generation = 1 req_id = 06ba02d9-679e-44ab-bd0b-595241cc144d
25032025-04-30T06:25:34.894ZDEBGtest_replacement_sanity (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 82724bc3-c920-4fde-a779-1ccedc0b0889 dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
25042025-04-30T06:25:34.894ZDEBGtest_replacement_sanity (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
25052025-04-30T06:25:34.894ZINFOtest_replacement_sanity (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
25062025-04-30T06:25:34.894ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 720.680909ms iteration = 2 status = {"generation":2}
25072025-04-30T06:25:34.894ZINFOtest_replacement_sanity (http): request completed kind = dns latency_us = 22711 local_addr = [::1]:34917 method = PUT remote_addr = [::1]:34158 req_id = 06ba02d9-679e-44ab-bd0b-595241cc144d response_code = 204 uri = /config
25082025-04-30T06:25:34.895ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
25092025-04-30T06:25:34.969ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:35427 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:35427] } uri = http://[::1]:35427/config
25102025-04-30T06:25:34.971ZINFOtest_replacement_sanity (http): accepted connection kind = dns local_addr = [::1]:35427 remote_addr = [::1]:46786
25112025-04-30T06:25:34.972ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
25122025-04-30T06:25:34.972ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
25132025-04-30T06:25:34.972ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
25142025-04-30T06:25:34.979ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:34917 generation = 1 result = Ok(Response { url: "http://[::1]:34917/config", status: 204, headers: {"x-request-id": "06ba02d9-679e-44ab-bd0b-595241cc144d", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} }) servers = DnsServersList { addresses: [[::1]:34917] }
25152025-04-30T06:25:34.979ZINFOtest_replacement_sanity (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:34917 generation = 1 servers = DnsServersList { addresses: [[::1]:34917] }
25162025-04-30T06:25:34.980ZINFOtest_replacement_sanity (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:34917] }
25172025-04-30T06:25:34.981ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 260.864064ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:34917":{"Ok":null}}}
25182025-04-30T06:25:34.982ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
25192025-04-30T06:25:34.982ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25202025-04-30T06:25:34.982ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25212025-04-30T06:25:34.983ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25222025-04-30T06:25:34.984ZTRACtest_replacement_sanity (http): incoming request kind = dns local_addr = [::1]:35427 method = PUT remote_addr = [::1]:46786 req_id = 99f2d5cc-ed34-4f74-89b4-e0ce56b0ef12 uri = /config
25232025-04-30T06:25:34.985ZINFOtest_replacement_sanity (store): attempting generation update new_generation = 2 req_id = 99f2d5cc-ed34-4f74-89b4-e0ce56b0ef12
25242025-04-30T06:25:34.985ZINFOtest_replacement_sanity (store): pruning trees for generations newer than 0
25252025-04-30T06:25:34.985ZDEBGtest_replacement_sanity (store): creating tree tree_name = generation_2_zone_oxide-dev.test
25262025-04-30T06:25:34.986ZDEBGtest_replacement_sanity (store): updating current config
25272025-04-30T06:25:34.986ZDEBGtest_replacement_sanity (store): flushing default tree
25282025-04-30T06:25:34.986ZINFOtest_replacement_sanity (store): pruning trees for generations older than 2 keep = 3
25292025-04-30T06:25:34.986ZINFOtest_replacement_sanity (store): updated generation new_generation = 2 req_id = 99f2d5cc-ed34-4f74-89b4-e0ce56b0ef12
25302025-04-30T06:25:34.987ZINFOtest_replacement_sanity (http): request completed kind = dns latency_us = 2505 local_addr = [::1]:35427 method = PUT remote_addr = [::1]:46786 req_id = 99f2d5cc-ed34-4f74-89b4-e0ce56b0ef12 response_code = 204 uri = /config
25312025-04-30T06:25:34.991ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:35427 generation = 2 result = Ok(Response { url: "http://[::1]:35427/config", status: 204, headers: {"x-request-id": "99f2d5cc-ed34-4f74-89b4-e0ce56b0ef12", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} }) servers = DnsServersList { addresses: [[::1]:35427] }
25322025-04-30T06:25:34.992ZINFOtest_replacement_sanity (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:35427 generation = 2 servers = DnsServersList { addresses: [[::1]:35427] }
25332025-04-30T06:25:35.001ZINFOtest_replacement_sanity (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:35427] }
25342025-04-30T06:25:35.001ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 106.553841ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:35427":{"Ok":null}}}
25352025-04-30T06:25:35.006ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25362025-04-30T06:25:35.013ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
25372025-04-30T06:25:35.013ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
25382025-04-30T06:25:35.013ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
25392025-04-30T06:25:35.013ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
25402025-04-30T06:25:35.013ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25412025-04-30T06:25:35.013ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25422025-04-30T06:25:35.014ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25432025-04-30T06:25:35.027ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc1106ad-8379-4a5c-8451-eacbe82f6f08", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
25442025-04-30T06:25:35.027ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
25452025-04-30T06:25:35.030ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
25462025-04-30T06:25:35.030ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25472025-04-30T06:25:35.030ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25482025-04-30T06:25:35.031ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25492025-04-30T06:25:35.039ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
25502025-04-30T06:25:35.039ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25512025-04-30T06:25:35.039ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25522025-04-30T06:25:35.040ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25532025-04-30T06:25:35.040ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5286c9f0-ca94-44c0-adad-9f9217453527", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
25542025-04-30T06:25:35.041ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
25552025-04-30T06:25:35.044ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25562025-04-30T06:25:35.051ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
25572025-04-30T06:25:35.051ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25582025-04-30T06:25:35.051ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25592025-04-30T06:25:35.052ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25602025-04-30T06:25:35.054ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5edcfc87-df7b-4f05-9c27-309588e7404b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
25612025-04-30T06:25:35.055ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
25622025-04-30T06:25:35.056ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25632025-04-30T06:25:35.056ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25642025-04-30T06:25:35.057ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25652025-04-30T06:25:35.060ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3e3bc53-6496-4793-85e7-53102c534534", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
25662025-04-30T06:25:35.060ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
25672025-04-30T06:25:35.061ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25682025-04-30T06:25:35.061ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25692025-04-30T06:25:35.062ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25702025-04-30T06:25:35.078ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05f74622-c387-46eb-bbd8-90ea1b95a499", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
25712025-04-30T06:25:35.078ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
25722025-04-30T06:25:35.079ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25732025-04-30T06:25:35.079ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25742025-04-30T06:25:35.090ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25752025-04-30T06:25:35.092ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
25762025-04-30T06:25:35.092ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
25772025-04-30T06:25:35.092ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25782025-04-30T06:25:35.092ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25792025-04-30T06:25:35.093ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25802025-04-30T06:25:35.097ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
25812025-04-30T06:25:35.097ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25822025-04-30T06:25:35.097ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25832025-04-30T06:25:35.098ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25842025-04-30T06:25:35.099ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c302caf-120e-4ef1-8e10-8d75d0c09b27", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
25852025-04-30T06:25:35.100ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
25862025-04-30T06:25:35.127ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
25872025-04-30T06:25:35.127ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25882025-04-30T06:25:35.127ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25892025-04-30T06:25:35.128ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25902025-04-30T06:25:35.154ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
25912025-04-30T06:25:35.161ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
25922025-04-30T06:25:35.161ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
25932025-04-30T06:25:35.161ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25942025-04-30T06:25:35.161ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25952025-04-30T06:25:35.162ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25962025-04-30T06:25:35.167ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
25972025-04-30T06:25:35.167ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
25982025-04-30T06:25:35.167ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
25992025-04-30T06:25:35.171ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26002025-04-30T06:25:35.171ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26012025-04-30T06:25:35.171ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26022025-04-30T06:25:35.185ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26032025-04-30T06:25:35.185ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26042025-04-30T06:25:35.186ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26052025-04-30T06:25:35.195ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03b5d767-3179-4d12-bb2b-b55ba2921784", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26062025-04-30T06:25:35.195ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
26072025-04-30T06:25:35.195ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
26082025-04-30T06:25:35.195ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
26092025-04-30T06:25:35.195ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26102025-04-30T06:25:35.195ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26112025-04-30T06:25:35.196ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26122025-04-30T06:25:35.202ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
26132025-04-30T06:25:35.202ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26142025-04-30T06:25:35.202ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26152025-04-30T06:25:35.205ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26162025-04-30T06:25:35.215ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
26172025-04-30T06:25:35.215ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26182025-04-30T06:25:35.215ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26192025-04-30T06:25:35.216ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26202025-04-30T06:25:35.255ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26212025-04-30T06:25:35.262ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
26222025-04-30T06:25:35.262ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
26232025-04-30T06:25:35.262ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
26242025-04-30T06:25:35.262ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
26252025-04-30T06:25:35.262ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26262025-04-30T06:25:35.262ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26272025-04-30T06:25:35.263ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26282025-04-30T06:25:35.267ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
26292025-04-30T06:25:35.267ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26302025-04-30T06:25:35.276ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26312025-04-30T06:25:35.277ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26322025-04-30T06:25:35.277ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "caa56069-6c38-40c7-9956-0e7970857cdd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26332025-04-30T06:25:35.277ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
26342025-04-30T06:25:35.279ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cda6702-f40d-4432-8cd2-15861bc5b3e7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26352025-04-30T06:25:35.280ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
26362025-04-30T06:25:35.280ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
26372025-04-30T06:25:35.280ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26382025-04-30T06:25:35.280ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26392025-04-30T06:25:35.281ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26402025-04-30T06:25:35.283ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "046969bc-c3fa-476c-948c-ecfd7807f7be", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26412025-04-30T06:25:35.283ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
26422025-04-30T06:25:35.296ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9582dd86-04fa-4190-8269-72228307e3be", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26432025-04-30T06:25:35.316ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
26442025-04-30T06:25:35.327ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26452025-04-30T06:25:35.342ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
26462025-04-30T06:25:35.342ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26472025-04-30T06:25:35.342ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26482025-04-30T06:25:35.343ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26492025-04-30T06:25:35.348ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75072930-df10-4fc4-8544-a272ee9ee861", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26502025-04-30T06:25:35.348ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
26512025-04-30T06:25:35.350ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26522025-04-30T06:25:35.357ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26532025-04-30T06:25:35.358ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26542025-04-30T06:25:35.359ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77064889-f482-4da5-a4f8-b66e4fe96f0d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26552025-04-30T06:25:35.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
26562025-04-30T06:25:35.366ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26572025-04-30T06:25:35.366ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26582025-04-30T06:25:35.368ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26592025-04-30T06:25:35.392ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26602025-04-30T06:25:35.392ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26612025-04-30T06:25:35.393ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26622025-04-30T06:25:35.403ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26632025-04-30T06:25:35.404ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26642025-04-30T06:25:35.404ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26652025-04-30T06:25:35.434ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
26662025-04-30T06:25:35.439ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
26672025-04-30T06:25:35.443ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26682025-04-30T06:25:35.443ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26692025-04-30T06:25:35.444ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26702025-04-30T06:25:35.444ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb5c72fd-a114-4421-a67c-3c33e9fda911", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26712025-04-30T06:25:35.445ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
26722025-04-30T06:25:35.458ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
26732025-04-30T06:25:35.458ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26742025-04-30T06:25:35.458ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26752025-04-30T06:25:35.458ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26762025-04-30T06:25:35.467ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
26772025-04-30T06:25:35.473ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26782025-04-30T06:25:35.473ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26792025-04-30T06:25:35.474ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26802025-04-30T06:25:35.480ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
26812025-04-30T06:25:35.524ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
26822025-04-30T06:25:35.524ZTRACtest_replacement_sanity (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
26832025-04-30T06:25:35.524ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
26842025-04-30T06:25:35.524ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
26852025-04-30T06:25:35.524ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26862025-04-30T06:25:35.524ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26872025-04-30T06:25:35.530ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26882025-04-30T06:25:35.560ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
26892025-04-30T06:25:35.563ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26902025-04-30T06:25:35.563ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26912025-04-30T06:25:35.564ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26922025-04-30T06:25:35.609ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7511c438-2c7a-4712-9510-3ddfd51caa66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26932025-04-30T06:25:35.609ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
26942025-04-30T06:25:35.613ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
26952025-04-30T06:25:35.613ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
26962025-04-30T06:25:35.614ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
26972025-04-30T06:25:35.614ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
26982025-04-30T06:25:35.616ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b92943ff-5e08-49cf-ae91-659662dbeaf6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
26992025-04-30T06:25:35.617ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
27002025-04-30T06:25:35.617ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27012025-04-30T06:25:35.628ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
27022025-04-30T06:25:35.629ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
27032025-04-30T06:25:35.629ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
27042025-04-30T06:25:35.631ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
27052025-04-30T06:25:35.639ZDEBGtest_replacement_sanity (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: 75d58370-2da0-4751-8dd3-8f5d02e520b7, kind: Service { id: c31da0ae-930d-4dd9-a897-159d5a6cd420 }, name: Name("nexus-c31da0ae-930d-4dd9-a897-159d5a6cd420"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 6ee0a63c-7d7b-4d03-a405-8d41adf7f5a5, kind: Service { id: 3ae50786-e3ad-43a3-866e-b9ad16af275d }, name: Name("external-dns-3ae50786-e3ad-43a3-866e-b9ad16af275d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
27062025-04-30T06:25:35.639ZTRACtest_replacement_sanity (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
27072025-04-30T06:25:35.639ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
27082025-04-30T06:25:35.639ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
27092025-04-30T06:25:35.639ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
27102025-04-30T06:25:35.640ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
27112025-04-30T06:25:35.642ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "394813be-d429-4c79-b3e5-82ba990ac37b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27122025-04-30T06:25:35.642ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
27132025-04-30T06:25:35.647ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27142025-04-30T06:25:35.652ZDEBGtest_replacement_sanity (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
27152025-04-30T06:25:35.652ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
27162025-04-30T06:25:35.652ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
27172025-04-30T06:25:35.653ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
27182025-04-30T06:25:35.655ZTRACtest_replacement_sanity (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:25:32.784618Z, allowed_ips: None }
27192025-04-30T06:25:35.655ZDEBGtest_replacement_sanity (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
27202025-04-30T06:25:35.656ZDEBGtest_replacement_sanity (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 6ee0a63c-7d7b-4d03-a405-8d41adf7f5a5, kind: Service { id: 3ae50786-e3ad-43a3-866e-b9ad16af275d }, name: Name("external-dns-3ae50786-e3ad-43a3-866e-b9ad16af275d"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 75d58370-2da0-4751-8dd3-8f5d02e520b7, kind: Service { id: c31da0ae-930d-4dd9-a897-159d5a6cd420 }, name: Name("nexus-c31da0ae-930d-4dd9-a897-159d5a6cd420"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
27212025-04-30T06:25:35.656ZDEBGtest_replacement_sanity (InternalApi): resolved 2 rules for sleds
27222025-04-30T06:25:35.658ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3b11756-7136-4b9f-8e7e-a5b7df7a351a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27232025-04-30T06:25:35.658ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
27242025-04-30T06:25:35.661ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd4bb765-ada7-42a0-8b00-fd6a1d6b36a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27252025-04-30T06:25:35.662ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
27262025-04-30T06:25:35.663ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d508f6-191f-4a8d-9c78-d2221649af48", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27272025-04-30T06:25:35.663ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
27282025-04-30T06:25:35.681ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "318fa2f8-e5e7-474e-bbe9-23f1834895f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27292025-04-30T06:25:35.681ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
27302025-04-30T06:25:35.683ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2632ce82-05ed-4dec-9fc2-0fb09b590440", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27312025-04-30T06:25:35.683ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
27322025-04-30T06:25:35.683ZDEBGtest_replacement_sanity (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:25:25.395837Z, time_modified: 2025-04-30T06:25:25.395837Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(35343), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(46651) }]
27332025-04-30T06:25:35.683ZDEBGtest_replacement_sanity (InternalApi): sending firewall rules to sled agents
27342025-04-30T06:25:35.683ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
27352025-04-30T06:25:35.684ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
27362025-04-30T06:25:35.684ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
27372025-04-30T06:25:35.689ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
27382025-04-30T06:25:35.689ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
27392025-04-30T06:25:35.689ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
27402025-04-30T06:25:35.689ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
27412025-04-30T06:25:35.690ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
27422025-04-30T06:25:35.690ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eccd75f6-8827-4e74-a51a-4a3a06914533", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27432025-04-30T06:25:35.691ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
27442025-04-30T06:25:35.693ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5104ccb-4f47-4b54-8748-2b57bab51cdd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27452025-04-30T06:25:35.693ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
27462025-04-30T06:25:35.693ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
27472025-04-30T06:25:35.698ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
27482025-04-30T06:25:35.783ZDEBGtest_replacement_sanity (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:35343/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27492025-04-30T06:25:35.783ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:43731 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
27502025-04-30T06:25:35.784ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:43731 req_id = e5f71129-ffc7-4910-96fd-31293abe10b0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27512025-04-30T06:25:35.785ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 803 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:43731 req_id = e5f71129-ffc7-4910-96fd-31293abe10b0 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
27522025-04-30T06:25:35.786ZDEBGtest_replacement_sanity (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:35343/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "e5f71129-ffc7-4910-96fd-31293abe10b0", "date": "Wed, 30 Apr 2025 06:25:34 GMT"} })
27532025-04-30T06:25:35.789ZINFOtest_replacement_sanity (InternalApi): updated firewall rules on 1 sleds
27542025-04-30T06:25:35.789ZINFOtest_replacement_sanity (nexus): plumbed initial IP allowlist
27552025-04-30T06:25:35.841ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = / versions = all
27562025-04-30T06:25:35.844ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /access versions = all
27572025-04-30T06:25:35.844ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /assets/{path} versions = all
27582025-04-30T06:25:35.844ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /device/auth versions = all
27592025-04-30T06:25:35.844ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /device/confirm versions = all
27602025-04-30T06:25:35.844ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /device/success versions = all
27612025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /device/token versions = all
27622025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /device/verify versions = all
27632025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/probes versions = all
27642025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /experimental/v1/probes versions = all
27652025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /experimental/v1/probes/{probe} versions = all
27662025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/probes/{probe} versions = all
27672025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/system/support-bundles versions = all
27682025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /experimental/v1/system/support-bundles versions = all
27692025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
27702025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
27712025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
27722025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
27732025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
27742025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
27752025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
27762025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /images versions = all
27772025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /login versions = all
27782025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /login/{silo_name}/local versions = all
27792025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
27802025-04-30T06:25:35.845ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
27812025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
27822025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /lookup/{path} versions = all
27832025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /projects/{path} versions = all
27842025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /projects-new versions = all
27852025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /settings/{path} versions = all
27862025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /system/{path} versions = all
27872025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /utilization versions = all
27882025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/affinity-groups versions = all
27892025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/affinity-groups versions = all
27902025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
27912025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
27922025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
27932025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
27942025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27952025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27962025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27972025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/anti-affinity-groups versions = all
27982025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/anti-affinity-groups versions = all
27992025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
28002025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
28012025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
28022025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
28032025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
28042025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
28052025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
28062025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/certificates versions = all
28072025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/certificates versions = all
28082025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/certificates/{certificate} versions = all
28092025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/certificates/{certificate} versions = all
28102025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/disks versions = all
28112025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/disks versions = all
28122025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/disks/{disk} versions = all
28132025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/disks/{disk} versions = all
28142025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/disks/{disk}/bulk-write versions = all
28152025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
28162025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
28172025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/disks/{disk}/finalize versions = all
28182025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
28192025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/floating-ips versions = all
28202025-04-30T06:25:35.846ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/floating-ips versions = all
28212025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
28222025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/floating-ips/{floating_ip} versions = all
28232025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
28242025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
28252025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
28262025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/groups versions = all
28272025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/groups/{group_id} versions = all
28282025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/images versions = all
28292025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/images versions = all
28302025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/images/{image} versions = all
28312025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/images/{image} versions = all
28322025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/images/{image}/demote versions = all
28332025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/images/{image}/promote versions = all
28342025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances versions = all
28352025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances versions = all
28362025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/instances/{instance} versions = all
28372025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance} versions = all
28382025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/instances/{instance} versions = all
28392025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
28402025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
28412025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/disks versions = all
28422025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances/{instance}/disks/attach versions = all
28432025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances/{instance}/disks/detach versions = all
28442025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/external-ips versions = all
28452025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
28462025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
28472025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances/{instance}/reboot versions = all
28482025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/serial-console versions = all
28492025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
28502025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
28512025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances/{instance}/start versions = all
28522025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/instances/{instance}/stop versions = all
28532025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/internet-gateway-ip-addresses versions = all
28542025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/internet-gateway-ip-addresses versions = all
28552025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
28562025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/internet-gateway-ip-pools versions = all
28572025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/internet-gateway-ip-pools versions = all
28582025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
28592025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/internet-gateways versions = all
28602025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/internet-gateways versions = all
28612025-04-30T06:25:35.847ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
28622025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/internet-gateways/{gateway} versions = all
28632025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/ip-pools versions = all
28642025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/ip-pools/{pool} versions = all
28652025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/login/{silo_name}/local versions = all
28662025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/logout versions = all
28672025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/me versions = all
28682025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/me/groups versions = all
28692025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/me/ssh-keys versions = all
28702025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/me/ssh-keys versions = all
28712025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
28722025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
28732025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/metrics/{metric_name} versions = all
28742025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/network-interfaces versions = all
28752025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/network-interfaces versions = all
28762025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/network-interfaces/{interface} versions = all
28772025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/network-interfaces/{interface} versions = all
28782025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/network-interfaces/{interface} versions = all
28792025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/ping versions = all
28802025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/policy versions = all
28812025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/policy versions = all
28822025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/projects versions = all
28832025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/projects versions = all
28842025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/projects/{project} versions = all
28852025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/projects/{project} versions = all
28862025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/projects/{project} versions = all
28872025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/projects/{project}/policy versions = all
28882025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/projects/{project}/policy versions = all
28892025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/snapshots versions = all
28902025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/snapshots versions = all
28912025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/snapshots/{snapshot} versions = all
28922025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/snapshots/{snapshot} versions = all
28932025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/disks versions = all
28942025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
28952025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
28962025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/racks versions = all
28972025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
28982025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/sleds versions = all
28992025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/hardware/sleds versions = all
29002025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
29012025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
29022025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
29032025-04-30T06:25:35.848ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
29042025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
29052025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/switch-port versions = all
29062025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
29072025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
29082025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
29092025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
29102025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
29112025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/switches versions = all
29122025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
29132025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/identity-providers versions = all
29142025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/identity-providers/local/users versions = all
29152025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
29162025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
29172025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/identity-providers/saml versions = all
29182025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
29192025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools versions = all
29202025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/ip-pools versions = all
29212025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
29222025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools/{pool} versions = all
29232025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/ip-pools/{pool} versions = all
29242025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
29252025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
29262025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
29272025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
29282025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
29292025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
29302025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
29312025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
29322025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools-service versions = all
29332025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/ip-pools-service/ranges versions = all
29342025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
29352025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
29362025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/metrics/{metric_name} versions = all
29372025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/address-lot versions = all
29382025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/networking/address-lot versions = all
29392025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
29402025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
29412025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/allow-list versions = all
29422025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/networking/allow-list versions = all
29432025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/networking/bfd-disable versions = all
29442025-04-30T06:25:35.849ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/networking/bfd-enable versions = all
29452025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bfd-status versions = all
29462025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/networking/bgp versions = all
29472025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp versions = all
29482025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/networking/bgp versions = all
29492025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp-announce-set versions = all
29502025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
29512025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
29522025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
29532025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp-exported versions = all
29542025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp-message-history versions = all
29552025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
29562025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/bgp-status versions = all
29572025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/loopback-address versions = all
29582025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/networking/loopback-address versions = all
29592025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
29602025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
29612025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/switch-port-settings versions = all
29622025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/networking/switch-port-settings versions = all
29632025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
29642025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/policy versions = all
29652025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/policy versions = all
29662025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/roles versions = all
29672025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/roles/{role_name} versions = all
29682025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/silo-quotas versions = all
29692025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/silos versions = all
29702025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/silos versions = all
29712025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/system/silos/{silo} versions = all
29722025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/silos/{silo} versions = all
29732025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
29742025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/silos/{silo}/policy versions = all
29752025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/silos/{silo}/policy versions = all
29762025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/silos/{silo}/quotas versions = all
29772025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
29782025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/system/timeseries/query versions = all
29792025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/timeseries/schemas versions = all
29802025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/update/repository versions = all
29812025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/update/repository/{system_version} versions = all
29822025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/update/target-release versions = all
29832025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/system/update/target-release versions = all
29842025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/users versions = all
29852025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/users/{user_id} versions = all
29862025-04-30T06:25:35.850ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/users-builtin versions = all
29872025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/users-builtin/{user} versions = all
29882025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/utilization/silos versions = all
29892025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/system/utilization/silos/{silo} versions = all
29902025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/timeseries/query versions = all
29912025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/users versions = all
29922025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/utilization versions = all
29932025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-firewall-rules versions = all
29942025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/vpc-firewall-rules versions = all
29952025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-router-routes versions = all
29962025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/vpc-router-routes versions = all
29972025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
29982025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-router-routes/{route} versions = all
29992025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/vpc-router-routes/{route} versions = all
30002025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-routers versions = all
30012025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/vpc-routers versions = all
30022025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/vpc-routers/{router} versions = all
30032025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-routers/{router} versions = all
30042025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/vpc-routers/{router} versions = all
30052025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-subnets versions = all
30062025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/vpc-subnets versions = all
30072025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
30082025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-subnets/{subnet} versions = all
30092025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
30102025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
30112025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpcs versions = all
30122025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/vpcs versions = all
30132025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/vpcs/{vpc} versions = all
30142025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/vpcs/{vpc} versions = all
30152025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/vpcs/{vpc} versions = all
30162025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/webhooks/deliveries versions = all
30172025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
30182025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/webhooks/event-classes versions = all
30192025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/webhooks/receivers versions = all
30202025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/webhooks/receivers versions = all
30212025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
30222025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
30232025-04-30T06:25:35.851ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
30242025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
30252025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
30262025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
30272025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = GET path = /v1/webhooks/secrets versions = all
30282025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = POST path = /v1/webhooks/secrets versions = all
30292025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): registered endpoint local_addr = 127.0.0.1:59162 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
30302025-04-30T06:25:35.852ZINFOtest_replacement_sanity (dropshot_external): listening local_addr = 127.0.0.1:59162
30312025-04-30T06:25:35.852ZDEBGtest_replacement_sanity (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:59162
30322025-04-30T06:25:35.909ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = / versions = all
30332025-04-30T06:25:35.909ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /access versions = all
30342025-04-30T06:25:35.909ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /assets/{path} versions = all
30352025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /device/auth versions = all
30362025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /device/confirm versions = all
30372025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /device/success versions = all
30382025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /device/token versions = all
30392025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /device/verify versions = all
30402025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/probes versions = all
30412025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /experimental/v1/probes versions = all
30422025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /experimental/v1/probes/{probe} versions = all
30432025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/probes/{probe} versions = all
30442025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/system/support-bundles versions = all
30452025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /experimental/v1/system/support-bundles versions = all
30462025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
30472025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
30482025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
30492025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
30502025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
30512025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
30522025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
30532025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /images versions = all
30542025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /login versions = all
30552025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /login/{silo_name}/local versions = all
30562025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
30572025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
30582025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
30592025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /lookup/{path} versions = all
30602025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /projects/{path} versions = all
30612025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /projects-new versions = all
30622025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /settings/{path} versions = all
30632025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /system/{path} versions = all
30642025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /utilization versions = all
30652025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/affinity-groups versions = all
30662025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/affinity-groups versions = all
30672025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
30682025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
30692025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
30702025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
30712025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
30722025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
30732025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
30742025-04-30T06:25:35.910ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/anti-affinity-groups versions = all
30752025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/anti-affinity-groups versions = all
30762025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
30772025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
30782025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
30792025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
30802025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
30812025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
30822025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
30832025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/certificates versions = all
30842025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/certificates versions = all
30852025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/certificates/{certificate} versions = all
30862025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/certificates/{certificate} versions = all
30872025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/disks versions = all
30882025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/disks versions = all
30892025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/disks/{disk} versions = all
30902025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/disks/{disk} versions = all
30912025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/disks/{disk}/bulk-write versions = all
30922025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
30932025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
30942025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/disks/{disk}/finalize versions = all
30952025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
30962025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/floating-ips versions = all
30972025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/floating-ips versions = all
30982025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
30992025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/floating-ips/{floating_ip} versions = all
31002025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
31012025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
31022025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
31032025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/groups versions = all
31042025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/groups/{group_id} versions = all
31052025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/images versions = all
31062025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/images versions = all
31072025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/images/{image} versions = all
31082025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/images/{image} versions = all
31092025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/images/{image}/demote versions = all
31102025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/images/{image}/promote versions = all
31112025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances versions = all
31122025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances versions = all
31132025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/instances/{instance} versions = all
31142025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance} versions = all
31152025-04-30T06:25:35.911ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/instances/{instance} versions = all
31162025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
31172025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
31182025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/disks versions = all
31192025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances/{instance}/disks/attach versions = all
31202025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances/{instance}/disks/detach versions = all
31212025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/external-ips versions = all
31222025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
31232025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
31242025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances/{instance}/reboot versions = all
31252025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/serial-console versions = all
31262025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
31272025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
31282025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances/{instance}/start versions = all
31292025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/instances/{instance}/stop versions = all
31302025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/internet-gateway-ip-addresses versions = all
31312025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/internet-gateway-ip-addresses versions = all
31322025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
31332025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/internet-gateway-ip-pools versions = all
31342025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/internet-gateway-ip-pools versions = all
31352025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
31362025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/internet-gateways versions = all
31372025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/internet-gateways versions = all
31382025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
31392025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/internet-gateways/{gateway} versions = all
31402025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/ip-pools versions = all
31412025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/ip-pools/{pool} versions = all
31422025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/login/{silo_name}/local versions = all
31432025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/logout versions = all
31442025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/me versions = all
31452025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/me/groups versions = all
31462025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/me/ssh-keys versions = all
31472025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/me/ssh-keys versions = all
31482025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
31492025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
31502025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/metrics/{metric_name} versions = all
31512025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/network-interfaces versions = all
31522025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/network-interfaces versions = all
31532025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/network-interfaces/{interface} versions = all
31542025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/network-interfaces/{interface} versions = all
31552025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/network-interfaces/{interface} versions = all
31562025-04-30T06:25:35.912ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/ping versions = all
31572025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/policy versions = all
31582025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/policy versions = all
31592025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/projects versions = all
31602025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/projects versions = all
31612025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/projects/{project} versions = all
31622025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/projects/{project} versions = all
31632025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/projects/{project} versions = all
31642025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/projects/{project}/policy versions = all
31652025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/projects/{project}/policy versions = all
31662025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/snapshots versions = all
31672025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/snapshots versions = all
31682025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/snapshots/{snapshot} versions = all
31692025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/snapshots/{snapshot} versions = all
31702025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/disks versions = all
31712025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
31722025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
31732025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/racks versions = all
31742025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
31752025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/sleds versions = all
31762025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/hardware/sleds versions = all
31772025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
31782025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
31792025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
31802025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
31812025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
31822025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/switch-port versions = all
31832025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
31842025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
31852025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
31862025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
31872025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
31882025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/switches versions = all
31892025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
31902025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/identity-providers versions = all
31912025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/identity-providers/local/users versions = all
31922025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
31932025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
31942025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/identity-providers/saml versions = all
31952025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
31962025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools versions = all
31972025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/ip-pools versions = all
31982025-04-30T06:25:35.913ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
31992025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools/{pool} versions = all
32002025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/ip-pools/{pool} versions = all
32012025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
32022025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
32032025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
32042025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
32052025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
32062025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
32072025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
32082025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
32092025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools-service versions = all
32102025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/ip-pools-service/ranges versions = all
32112025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
32122025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
32132025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/metrics/{metric_name} versions = all
32142025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/address-lot versions = all
32152025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/networking/address-lot versions = all
32162025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
32172025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
32182025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/allow-list versions = all
32192025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/networking/allow-list versions = all
32202025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/networking/bfd-disable versions = all
32212025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/networking/bfd-enable versions = all
32222025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bfd-status versions = all
32232025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/networking/bgp versions = all
32242025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp versions = all
32252025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/networking/bgp versions = all
32262025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp-announce-set versions = all
32272025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
32282025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
32292025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
32302025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp-exported versions = all
32312025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp-message-history versions = all
32322025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
32332025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/bgp-status versions = all
32342025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/loopback-address versions = all
32352025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/networking/loopback-address versions = all
32362025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
32372025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
32382025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/switch-port-settings versions = all
32392025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/networking/switch-port-settings versions = all
32402025-04-30T06:25:35.914ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
32412025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/policy versions = all
32422025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/policy versions = all
32432025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/roles versions = all
32442025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/roles/{role_name} versions = all
32452025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/silo-quotas versions = all
32462025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/silos versions = all
32472025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/silos versions = all
32482025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/system/silos/{silo} versions = all
32492025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/silos/{silo} versions = all
32502025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
32512025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/silos/{silo}/policy versions = all
32522025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/silos/{silo}/policy versions = all
32532025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/silos/{silo}/quotas versions = all
32542025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
32552025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/system/timeseries/query versions = all
32562025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/timeseries/schemas versions = all
32572025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/update/repository versions = all
32582025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/update/repository/{system_version} versions = all
32592025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/update/target-release versions = all
32602025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/system/update/target-release versions = all
32612025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/users versions = all
32622025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/users/{user_id} versions = all
32632025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/users-builtin versions = all
32642025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/users-builtin/{user} versions = all
32652025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/utilization/silos versions = all
32662025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/system/utilization/silos/{silo} versions = all
32672025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/timeseries/query versions = all
32682025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/users versions = all
32692025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/utilization versions = all
32702025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-firewall-rules versions = all
32712025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/vpc-firewall-rules versions = all
32722025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-router-routes versions = all
32732025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/vpc-router-routes versions = all
32742025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
32752025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-router-routes/{route} versions = all
32762025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/vpc-router-routes/{route} versions = all
32772025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-routers versions = all
32782025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/vpc-routers versions = all
32792025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/vpc-routers/{router} versions = all
32802025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-routers/{router} versions = all
32812025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/vpc-routers/{router} versions = all
32822025-04-30T06:25:35.915ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-subnets versions = all
32832025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/vpc-subnets versions = all
32842025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
32852025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-subnets/{subnet} versions = all
32862025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
32872025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
32882025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpcs versions = all
32892025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/vpcs versions = all
32902025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/vpcs/{vpc} versions = all
32912025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/vpcs/{vpc} versions = all
32922025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/vpcs/{vpc} versions = all
32932025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/webhooks/deliveries versions = all
32942025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
32952025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/webhooks/event-classes versions = all
32962025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/webhooks/receivers versions = all
32972025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/webhooks/receivers versions = all
32982025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
32992025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
33002025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
33012025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
33022025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
33032025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
33042025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = GET path = /v1/webhooks/secrets versions = all
33052025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = POST path = /v1/webhooks/secrets versions = all
33062025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:53387 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
33072025-04-30T06:25:35.916ZINFOtest_replacement_sanity (dropshot_external_techport): listening local_addr = 127.0.0.1:53387
33082025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:53387
33092025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (nexus-producer-server): registered DTrace probes
33102025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = 127.0.0.1:50502 method = GET path = /{producer_id} versions = all
33112025-04-30T06:25:35.916ZINFOtest_replacement_sanity (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:50502
33122025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:50502
33132025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:50502
33142025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:43527
33152025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (nexus-producer-server): starting producer registration task
33162025-04-30T06:25:35.916ZINFOtest_replacement_sanity (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:50502 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
33172025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (init_with_steps): Running step start_oximeter
33182025-04-30T06:25:35.916ZDEBGtest_replacement_sanity: Starting Oximeter
33192025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (oximeter): registered DTrace probes
33202025-04-30T06:25:35.916ZINFOtest_replacement_sanity (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
33212025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (oximeter): creating ClickHouse client
33222025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
33232025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
33242025-04-30T06:25:35.916ZDEBGtest_replacement_sanity (producer-registration-task): will register with Nexus at 127.0.0.1:43527
33252025-04-30T06:25:35.999ZDEBGtest_replacement_sanity (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:43527/metrics/producers
33262025-04-30T06:25:36.002ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:57098
33272025-04-30T06:25:36.004ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:57098 req_id = 4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a uri = /metrics/producers
33282025-04-30T06:25:36.006ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:57098 req_id = 4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a resource = Database uri = /metrics/producers
33292025-04-30T06:25:36.006ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:57098 req_id = 4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a roles = RoleSet { roles: {} } uri = /metrics/producers
33302025-04-30T06:25:36.007ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:57098 req_id = 4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a resource = Database result = Ok(()) uri = /metrics/producers
33312025-04-30T06:25:36.012ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
33322025-04-30T06:25:36.018ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "430d0797-37bf-4111-a94d-e4641bb2545a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33332025-04-30T06:25:36.018ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
33342025-04-30T06:25:36.021ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05bd527d-5d21-46b5-8d9e-a3eab0dbf34c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33352025-04-30T06:25:36.022ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
33362025-04-30T06:25:36.024ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc406b93-bbcc-4dd0-9e84-eac8d64d86fc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33372025-04-30T06:25:36.024ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
33382025-04-30T06:25:36.026ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3adf8863-3de8-4450-9b8a-c9bf436875de", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33392025-04-30T06:25:36.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
33402025-04-30T06:25:36.026ZWARNtest_replacement_sanity (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = b8acd62a-f986-4471-9673-45282afe5833
33412025-04-30T06:25:36.026ZDEBGtest_replacement_sanity (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
33422025-04-30T06:25:36.026ZINFOtest_replacement_sanity (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = b8acd62a-f986-4471-9673-45282afe5833
33432025-04-30T06:25:36.027ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT MAX(value) AS max_version FROM oximeter.version;
33442025-04-30T06:25:36.028ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ee5ac7-6b2e-4c77-a77b-3cc960a8a791", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33452025-04-30T06:25:36.028ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
33462025-04-30T06:25:36.028ZWARNtest_replacement_sanity (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = b8acd62a-f986-4471-9673-45282afe5833
33472025-04-30T06:25:36.028ZINFOtest_replacement_sanity (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = b8acd62a-f986-4471-9673-45282afe5833 version = 0
33482025-04-30T06:25:36.028ZINFOtest_replacement_sanity (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = b8acd62a-f986-4471-9673-45282afe5833
33492025-04-30T06:25:36.028ZDEBGtest_replacement_sanity (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
33502025-04-30T06:25:36.029ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = CREATE DATABASE IF NOT EXISTS oximeter
33512025-04-30T06:25:36.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecf14594-4914-460d-bef8-eac4d489a05e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33522025-04-30T06:25:36.030ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
33532025-04-30T06:25:36.031ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
33542025-04-30T06:25:36.031ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a1c11b8-b848-49e8-9e6c-b361ae405b2b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33552025-04-30T06:25:36.032ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
33562025-04-30T06:25:36.032ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7794744-bf2e-4e7c-9c7b-143dc38f5da2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33572025-04-30T06:25:36.033ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
33582025-04-30T06:25:36.033ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33592025-04-30T06:25:36.034ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "968a3402-03d3-4c98-982b-38ea9a328def", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33602025-04-30T06:25:36.034ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
33612025-04-30T06:25:36.035ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71fd2716-3ab3-4e5f-94be-ed9497439098", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33622025-04-30T06:25:36.035ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
33632025-04-30T06:25:36.036ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33642025-04-30T06:25:36.036ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b660110-1f89-4e06-a666-7cd25a064dc5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33652025-04-30T06:25:36.036ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
33662025-04-30T06:25:36.038ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33672025-04-30T06:25:36.040ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0be74008-f58b-4609-8f02-727fdf5e8550", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33682025-04-30T06:25:36.040ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
33692025-04-30T06:25:36.041ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33702025-04-30T06:25:36.044ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33712025-04-30T06:25:36.047ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33722025-04-30T06:25:36.050ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33732025-04-30T06:25:36.052ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33742025-04-30T06:25:36.055ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d80e5931-5ff8-4429-b78e-e773605e5682", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33752025-04-30T06:25:36.055ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
33762025-04-30T06:25:36.055ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33772025-04-30T06:25:36.058ZINFOtest_replacement_sanity (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 53987 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:57098 req_id = 4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a response_code = 503 uri = /metrics/producers
33782025-04-30T06:25:36.059ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33792025-04-30T06:25:36.061ZDEBGtest_replacement_sanity (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a", "content-length": "133", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33802025-04-30T06:25:36.062ZWARNtest_replacement_sanity (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 146.892353ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:25:36 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"4bff5b8e-83fe-4f29-b1e0-9f4ef16b119a\\" }" file = oximeter/producer/src/lib.rs:371
33812025-04-30T06:25:36.063ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33822025-04-30T06:25:36.065ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33832025-04-30T06:25:36.068ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33842025-04-30T06:25:36.071ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33852025-04-30T06:25:36.079ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33862025-04-30T06:25:36.087ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33872025-04-30T06:25:36.090ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33882025-04-30T06:25:36.096ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21dd09f4-6ff1-4130-baf5-f04a4f4ba872", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33892025-04-30T06:25:36.096ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
33902025-04-30T06:25:36.096ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33912025-04-30T06:25:36.100ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33922025-04-30T06:25:36.103ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33932025-04-30T06:25:36.107ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33942025-04-30T06:25:36.111ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33952025-04-30T06:25:36.113ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "850937e2-c8e1-4de9-92aa-f0dd86136001", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
33962025-04-30T06:25:36.113ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
33972025-04-30T06:25:36.114ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33982025-04-30T06:25:36.132ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
33992025-04-30T06:25:36.135ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86ca0c0f-6b8e-446c-bc80-6f1ae169cd3c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34002025-04-30T06:25:36.135ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
34012025-04-30T06:25:36.136ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34022025-04-30T06:25:36.138ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c1bc6a0-6bef-4f58-950e-d4398a8e5800", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34032025-04-30T06:25:36.138ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
34042025-04-30T06:25:36.139ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22628a69-2eac-4720-88b0-a87ecce58db8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34052025-04-30T06:25:36.139ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
34062025-04-30T06:25:36.140ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34072025-04-30T06:25:36.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "499a7e3d-c9d2-46d0-96e4-865ffd7717fb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34082025-04-30T06:25:36.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
34092025-04-30T06:25:36.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d366c9a-85e2-43dd-ba6e-d6b1a5dce3f7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34102025-04-30T06:25:36.142ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
34112025-04-30T06:25:36.144ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
34122025-04-30T06:25:36.145ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7529870e-f678-442c-95c4-ad9ac47d2064", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34132025-04-30T06:25:36.145ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
34142025-04-30T06:25:36.145ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
34152025-04-30T06:25:36.151ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34162025-04-30T06:25:36.161ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb325b23-6227-4e32-b512-64dec53d1566", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34172025-04-30T06:25:36.161ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
34182025-04-30T06:25:36.165ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34192025-04-30T06:25:36.168ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "18476dd3-948c-4a94-9e5a-1f17d9deae91", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34202025-04-30T06:25:36.168ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
34212025-04-30T06:25:36.178ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34222025-04-30T06:25:36.178ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "884ca8ad-b1ac-4458-81c7-944c33241952", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34232025-04-30T06:25:36.178ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
34242025-04-30T06:25:36.180ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "63869e06-da5d-4b77-bffc-fdf30674689e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34252025-04-30T06:25:36.180ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
34262025-04-30T06:25:36.181ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e4564be-807e-4b57-92a8-fb78d7254b2b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34272025-04-30T06:25:36.182ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
34282025-04-30T06:25:36.182ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34292025-04-30T06:25:36.182ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34302025-04-30T06:25:36.182ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34312025-04-30T06:25:36.184ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34322025-04-30T06:25:36.188ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34332025-04-30T06:25:36.192ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
34342025-04-30T06:25:36.192ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
34352025-04-30T06:25:36.192ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34362025-04-30T06:25:36.192ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34372025-04-30T06:25:36.192ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34382025-04-30T06:25:36.193ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34392025-04-30T06:25:36.193ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34402025-04-30T06:25:36.201ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34412025-04-30T06:25:36.201ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34422025-04-30T06:25:36.207ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
34432025-04-30T06:25:36.207ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34442025-04-30T06:25:36.207ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34452025-04-30T06:25:36.207ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34462025-04-30T06:25:36.214ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34472025-04-30T06:25:36.301ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
34482025-04-30T06:25:36.301ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
34492025-04-30T06:25:36.302ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34502025-04-30T06:25:36.305ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:63380 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
34512025-04-30T06:25:36.307ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34522025-04-30T06:25:36.309ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:63380 req_id = 691ccaeb-d3e0-440a-a302-3bc95c28bbb8 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
34532025-04-30T06:25:36.311ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1606 local_addr = [::1]:42668 method = GET remote_addr = [::1]:63380 req_id = 691ccaeb-d3e0-440a-a302-3bc95c28bbb8 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
34542025-04-30T06:25:36.312ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34552025-04-30T06:25:36.314ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "691ccaeb-d3e0-440a-a302-3bc95c28bbb8", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34562025-04-30T06:25:36.382ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
34572025-04-30T06:25:36.382ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
34582025-04-30T06:25:36.382ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34592025-04-30T06:25:36.382ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:48604 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
34602025-04-30T06:25:36.383ZDEBGtest_replacement_sanity (producer-registration-task): will register with Nexus at 127.0.0.1:43527
34612025-04-30T06:25:36.444ZDEBGtest_replacement_sanity (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:43527/metrics/producers
34622025-04-30T06:25:36.445ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34632025-04-30T06:25:36.445ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:52375
34642025-04-30T06:25:36.446ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:48604 req_id = aa31808a-09ed-4af4-8bbf-902bea1298bf sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
34652025-04-30T06:25:36.447ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1037 local_addr = [::1]:47918 method = GET remote_addr = [::1]:48604 req_id = aa31808a-09ed-4af4-8bbf-902bea1298bf response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
34662025-04-30T06:25:36.448ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:52375 req_id = d023b6ae-d714-4a6f-a0d6-406def554535 uri = /metrics/producers
34672025-04-30T06:25:36.448ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa31808a-09ed-4af4-8bbf-902bea1298bf", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34682025-04-30T06:25:36.503ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
34692025-04-30T06:25:36.503ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
34702025-04-30T06:25:36.505ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:52375 req_id = d023b6ae-d714-4a6f-a0d6-406def554535 resource = Database uri = /metrics/producers
34712025-04-30T06:25:36.505ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:52375 req_id = d023b6ae-d714-4a6f-a0d6-406def554535 roles = RoleSet { roles: {} } uri = /metrics/producers
34722025-04-30T06:25:36.506ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:52375 req_id = d023b6ae-d714-4a6f-a0d6-406def554535 resource = Database result = Ok(()) uri = /metrics/producers
34732025-04-30T06:25:36.507ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
34742025-04-30T06:25:36.508ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:59542 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
34752025-04-30T06:25:36.509ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:59542 req_id = eafeac7a-c990-4feb-8f34-c3d1c1aef5f9 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
34762025-04-30T06:25:36.510ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 338 local_addr = [::1]:35744 method = GET remote_addr = [::1]:59542 req_id = eafeac7a-c990-4feb-8f34-c3d1c1aef5f9 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
34772025-04-30T06:25:36.516ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "eafeac7a-c990-4feb-8f34-c3d1c1aef5f9", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34782025-04-30T06:25:36.577ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
34792025-04-30T06:25:36.578ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
34802025-04-30T06:25:36.579ZINFOtest_replacement_sanity (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 131201 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:52375 req_id = d023b6ae-d714-4a6f-a0d6-406def554535 response_code = 503 uri = /metrics/producers
34812025-04-30T06:25:36.580ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
34822025-04-30T06:25:36.580ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:34293 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
34832025-04-30T06:25:36.580ZDEBGtest_replacement_sanity (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "d023b6ae-d714-4a6f-a0d6-406def554535", "content-length": "133", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34842025-04-30T06:25:36.582ZWARNtest_replacement_sanity (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 666.069741ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"d023b6ae-d714-4a6f-a0d6-406def554535\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:25:36 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"d023b6ae-d714-4a6f-a0d6-406def554535\\" }" file = oximeter/producer/src/lib.rs:371
34852025-04-30T06:25:36.583ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:34293 req_id = beefd497-bcdc-4302-8a73-8c5dfd6bb598 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
34862025-04-30T06:25:36.588ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 5477 local_addr = [::1]:35343 method = GET remote_addr = [::1]:34293 req_id = beefd497-bcdc-4302-8a73-8c5dfd6bb598 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
34872025-04-30T06:25:36.588ZINFOtest_replacement_sanity (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = b8acd62a-f986-4471-9673-45282afe5833 version = 14
34882025-04-30T06:25:36.588ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = INSERT INTO oximeter.version (*) VALUES (14, now());
34892025-04-30T06:25:36.590ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "beefd497-bcdc-4302-8a73-8c5dfd6bb598", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:36 GMT"} })
34902025-04-30T06:25:36.594ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
34912025-04-30T06:25:36.594ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
34922025-04-30T06:25:36.594ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
34932025-04-30T06:25:36.597ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34942025-04-30T06:25:36.597ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34952025-04-30T06:25:36.597ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34962025-04-30T06:25:36.597ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34972025-04-30T06:25:36.598ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34982025-04-30T06:25:36.600ZDEBGtest_replacement_sanity (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 2cff5f08-d470-4262-aadc-5276f6e854e6
34992025-04-30T06:25:36.602ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:44431 method = GET path = /info versions = all
35002025-04-30T06:25:36.602ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:44431 method = GET path = /producers versions = all
35012025-04-30T06:25:36.602ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:44431 method = DELETE path = /producers/{producer_id} versions = all
35022025-04-30T06:25:36.602ZDEBGtest_replacement_sanity (dropshot): registered endpoint local_addr = [::1]:44431 method = GET path = /producers/{producer_id} versions = all
35032025-04-30T06:25:36.602ZINFOtest_replacement_sanity (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:44431
35042025-04-30T06:25:36.602ZDEBGtest_replacement_sanity (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:44431
35052025-04-30T06:25:36.604ZDEBGtest_replacement_sanity (oximeter): registered USDT probes
35062025-04-30T06:25:36.605ZDEBGtest_replacement_sanity (oximeter): contacting nexus
35072025-04-30T06:25:37.186ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35082025-04-30T06:25:37.189ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
35092025-04-30T06:25:37.189ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35102025-04-30T06:25:37.190ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35112025-04-30T06:25:37.198ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35122025-04-30T06:25:37.199ZDEBGtest_replacement_sanity (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:43527/metrics/collectors
35132025-04-30T06:25:37.201ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:64764
35142025-04-30T06:25:37.219ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:64764 req_id = 244ee769-9a4c-4e0d-b1a0-f05b7e3dd284 uri = /metrics/collectors
35152025-04-30T06:25:37.222ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:64764 req_id = 244ee769-9a4c-4e0d-b1a0-f05b7e3dd284 resource = Database uri = /metrics/collectors
35162025-04-30T06:25:37.222ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:64764 req_id = 244ee769-9a4c-4e0d-b1a0-f05b7e3dd284 roles = RoleSet { roles: {} } uri = /metrics/collectors
35172025-04-30T06:25:37.223ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:64764 req_id = 244ee769-9a4c-4e0d-b1a0-f05b7e3dd284 resource = Database result = Ok(()) uri = /metrics/collectors
35182025-04-30T06:25:37.237ZINFOtest_replacement_sanity (nexus): registered new oximeter metric collection server address = [::1]:44431 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
35192025-04-30T06:25:37.237ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 18599 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:64764 req_id = 244ee769-9a4c-4e0d-b1a0-f05b7e3dd284 response_code = 204 uri = /metrics/collectors
35202025-04-30T06:25:37.248ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors", status: 204, headers: {"x-request-id": "244ee769-9a4c-4e0d-b1a0-f05b7e3dd284", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
35212025-04-30T06:25:37.248ZINFOtest_replacement_sanity (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
35222025-04-30T06:25:37.248ZDEBGtest_replacement_sanity (init_with_steps): Running step start_producer_server
35232025-04-30T06:25:37.248ZDEBGtest_replacement_sanity: Starting test metric Producer Server
35242025-04-30T06:25:37.252ZINFOtest_replacement_sanity: registering Oso class class = Action
35252025-04-30T06:25:37.252ZINFOtest_replacement_sanity: registering Oso class class = AnyActor
35262025-04-30T06:25:37.252ZINFOtest_replacement_sanity: registering Oso class class = AuthenticatedActor
35272025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = BlueprintConfig
35282025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Database
35292025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = DnsConfig
35302025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Fleet
35312025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Inventory
35322025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = IpPoolList
35332025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = ConsoleSessionList
35342025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = DeviceAuthRequestList
35352025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = SiloCertificateList
35362025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = SiloIdentityProviderList
35372025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = SiloUserList
35382025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = TargetReleaseConfig
35392025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = WebhookEventClassList
35402025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Project
35412025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Disk
35422025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Snapshot
35432025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = ProjectImage
35442025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = AffinityGroup
35452025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = AntiAffinityGroup
35462025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Instance
35472025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = IpPool
35482025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = InstanceNetworkInterface
35492025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Vpc
35502025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = VpcRouter
35512025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = InternetGateway
35522025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = InternetGatewayIpPool
35532025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = InternetGatewayIpAddress
35542025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = RouterRoute
35552025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = VpcSubnet
35562025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = FloatingIp
35572025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Image
35582025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = SiloImage
35592025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = AddressLot
35602025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = Blueprint
35612025-04-30T06:25:37.253ZINFOtest_replacement_sanity: registering Oso class class = LoopbackAddress
35622025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = Certificate
35632025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = ConsoleSession
35642025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = DeviceAuthRequest
35652025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = DeviceAccessToken
35662025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = PhysicalDisk
35672025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = Rack
35682025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = RoleBuiltin
35692025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = SshKey
35702025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = Silo
35712025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = SiloUser
35722025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = SiloGroup
35732025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = SupportBundle
35742025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = IdentityProvider
35752025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = SamlIdentityProvider
35762025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = Sled
35772025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = TufRepo
35782025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = TufArtifact
35792025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = WebhookEvent
35802025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = WebhookReceiver
35812025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = WebhookSecret
35822025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = Zpool
35832025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = Service
35842025-04-30T06:25:37.254ZINFOtest_replacement_sanity: registering Oso class class = UserBuiltin
35852025-04-30T06:25:37.254ZINFOtest_replacement_sanity: full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
35862025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = GET path = /crucible/0/regions server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35872025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = POST path = /crucible/0/regions server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35882025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = DELETE path = /crucible/0/regions/{id} server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35892025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = GET path = /crucible/0/regions/{id} server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35902025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = GET path = /crucible/0/regions/{id}/snapshots server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35912025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35922025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35932025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35942025-04-30T06:25:37.291ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:41097 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 versions = all
35952025-04-30T06:25:37.291ZINFOtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:41097 server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
35962025-04-30T06:25:37.294ZDEBGtest_replacement_sanity (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:41097 server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
35972025-04-30T06:25:37.294ZINFOtest_replacement_sanity (SledAgent): Created Simulated Crucible Server address = [::1]:41097 kind = storage server = 039be560-54cc-49e3-88df-1a29dadbf913 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
35982025-04-30T06:25:37.294ZINFOtest_replacement_sanity (nexus): inserting test physical disk model = test-model physical_disk_id = 94b61fd7-d18d-4188-b481-93469afb5006 serial = totally-unique-serial: 94b61fd7-d18d-4188-b481-93469afb5006 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 vendor = test-vendor
35992025-04-30T06:25:37.294ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36002025-04-30T06:25:37.294ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36012025-04-30T06:25:37.295ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36022025-04-30T06:25:37.297ZINFOtest_replacement_sanity (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
36032025-04-30T06:25:37.430ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:40128
36042025-04-30T06:25:37.430ZDEBGtest_replacement_sanity (producer-registration-task): will register with Nexus at 127.0.0.1:43527
36052025-04-30T06:25:37.539ZDEBGtest_replacement_sanity (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:43527/metrics/producers
36062025-04-30T06:25:37.540ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
36072025-04-30T06:25:37.545ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36082025-04-30T06:25:37.547ZINFOtest_replacement_sanity (dropshot_internal): accepted connection local_addr = 127.0.0.1:43527 remote_addr = 127.0.0.1:55566
36092025-04-30T06:25:37.547ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2c7e17ef-0793-4387-bec2-17d89b0fc8b6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36102025-04-30T06:25:37.547ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40128 req_id = affbc0ef-860b-41ec-ad0e-d4383a69dbff uri = /metrics/producers
36112025-04-30T06:25:37.550ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2c7e17ef-0793-4387-bec2-17d89b0fc8b6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36122025-04-30T06:25:37.550ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2c7e17ef-0793-4387-bec2-17d89b0fc8b6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36132025-04-30T06:25:37.550ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2c7e17ef-0793-4387-bec2-17d89b0fc8b6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36142025-04-30T06:25:37.551ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40128 req_id = affbc0ef-860b-41ec-ad0e-d4383a69dbff resource = Database uri = /metrics/producers
36152025-04-30T06:25:37.551ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40128 req_id = affbc0ef-860b-41ec-ad0e-d4383a69dbff roles = RoleSet { roles: {} } uri = /metrics/producers
36162025-04-30T06:25:37.551ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40128 req_id = affbc0ef-860b-41ec-ad0e-d4383a69dbff resource = Database result = Ok(()) uri = /metrics/producers
36172025-04-30T06:25:37.553ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36182025-04-30T06:25:37.553ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36192025-04-30T06:25:37.553ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36202025-04-30T06:25:37.554ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:55566 req_id = fd5104ef-0369-417c-af24-6db3dc06e2b9 uri = /metrics/producers
36212025-04-30T06:25:37.557ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:55566 req_id = fd5104ef-0369-417c-af24-6db3dc06e2b9 resource = Database uri = /metrics/producers
36222025-04-30T06:25:37.557ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:55566 req_id = fd5104ef-0369-417c-af24-6db3dc06e2b9 roles = RoleSet { roles: {} } uri = /metrics/producers
36232025-04-30T06:25:37.558ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:55566 req_id = fd5104ef-0369-417c-af24-6db3dc06e2b9 resource = Database result = Ok(()) uri = /metrics/producers
36242025-04-30T06:25:37.567ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 20373 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2c7e17ef-0793-4387-bec2-17d89b0fc8b6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
36252025-04-30T06:25:37.573ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
36262025-04-30T06:25:37.573ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36272025-04-30T06:25:37.573ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36282025-04-30T06:25:37.573ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36292025-04-30T06:25:37.574ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36302025-04-30T06:25:37.578ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c7e17ef-0793-4387-bec2-17d89b0fc8b6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
36312025-04-30T06:25:37.579ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
36322025-04-30T06:25:37.583ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36332025-04-30T06:25:37.587ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
36342025-04-30T06:25:37.619ZINFOtest_replacement_sanity (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
36352025-04-30T06:25:37.626ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 78434 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:40128 req_id = affbc0ef-860b-41ec-ad0e-d4383a69dbff response_code = 201 uri = /metrics/producers
36362025-04-30T06:25:37.627ZTRACtest_replacement_sanity (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
36372025-04-30T06:25:37.627ZTRACtest_replacement_sanity (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
36382025-04-30T06:25:37.641ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36392025-04-30T06:25:37.641ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36402025-04-30T06:25:37.641ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 3.466613321s iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36412025-04-30T06:25:37.641ZINFOtest_replacement_sanity (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
36422025-04-30T06:25:37.642ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 87159 local_addr = 127.0.0.1:43527 method = POST remote_addr = 127.0.0.1:55566 req_id = fd5104ef-0369-417c-af24-6db3dc06e2b9 response_code = 201 uri = /metrics/producers
36432025-04-30T06:25:37.646ZDEBGtest_replacement_sanity (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "fd5104ef-0369-417c-af24-6db3dc06e2b9", "content-length": "41", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
36442025-04-30T06:25:37.648ZDEBGtest_replacement_sanity (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
36452025-04-30T06:25:37.743ZINFOtest_replacement_sanity (nexus): upserting zpool physical_disk_id = 94b61fd7-d18d-4188-b481-93469afb5006 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 zpool_id = 09b3ae18-5751-4b07-bcba-9f2d85125228
36462025-04-30T06:25:37.743ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36472025-04-30T06:25:37.743ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36482025-04-30T06:25:37.744ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36492025-04-30T06:25:37.754ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
36502025-04-30T06:25:37.754ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36512025-04-30T06:25:37.754ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36522025-04-30T06:25:37.754ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36532025-04-30T06:25:37.755ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36542025-04-30T06:25:37.760ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36552025-04-30T06:25:37.763ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
36562025-04-30T06:25:37.775ZINFOtest_replacement_sanity (nexus): upserting Crucible dataset dataset_id = c69ce87a-a069-4ae1-b178-fe487c6ade1e zpool_id = 09b3ae18-5751-4b07-bcba-9f2d85125228
36572025-04-30T06:25:37.805ZINFOtest_replacement_sanity (nexus): inserting test physical disk model = test-model physical_disk_id = 94b61fd7-d18d-4188-b481-93469afb5006 serial = totally-unique-serial: 94b61fd7-d18d-4188-b481-93469afb5006 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 vendor = test-vendor
36582025-04-30T06:25:37.805ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36592025-04-30T06:25:37.805ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36602025-04-30T06:25:37.806ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36612025-04-30T06:25:37.820ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 94b61fd7-d18d-4188-b481-93469afb5006 (physical_disk), lookup_type: ById(94b61fd7-d18d-4188-b481-93469afb5006) }
36622025-04-30T06:25:37.820ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36632025-04-30T06:25:37.820ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36642025-04-30T06:25:37.820ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36652025-04-30T06:25:37.821ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36662025-04-30T06:25:37.826ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36672025-04-30T06:25:37.836ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 94b61fd7-d18d-4188-b481-93469afb5006 (physical_disk), lookup_type: ById(94b61fd7-d18d-4188-b481-93469afb5006) } result = Ok(())
36682025-04-30T06:25:37.836ZINFOtest_replacement_sanity (nexus): upserting zpool physical_disk_id = 94b61fd7-d18d-4188-b481-93469afb5006 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 zpool_id = 09b3ae18-5751-4b07-bcba-9f2d85125228
36692025-04-30T06:25:37.836ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36702025-04-30T06:25:37.836ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36712025-04-30T06:25:37.837ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36722025-04-30T06:25:37.838ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
36732025-04-30T06:25:37.838ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36742025-04-30T06:25:37.838ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36752025-04-30T06:25:37.838ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36762025-04-30T06:25:37.839ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36772025-04-30T06:25:37.841ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36782025-04-30T06:25:37.843ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
36792025-04-30T06:25:37.854ZTRACtest_replacement_sanity (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
36802025-04-30T06:25:37.855ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36812025-04-30T06:25:37.855ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36822025-04-30T06:25:37.855ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36832025-04-30T06:25:37.855ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36842025-04-30T06:25:37.872ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36852025-04-30T06:25:37.883ZDEBGtest_replacement_sanity (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
36862025-04-30T06:25:37.883ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36872025-04-30T06:25:37.883ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36882025-04-30T06:25:37.883ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36892025-04-30T06:25:37.894ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36902025-04-30T06:25:37.901ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36912025-04-30T06:25:37.902ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36922025-04-30T06:25:37.902ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
36932025-04-30T06:25:37.903ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36942025-04-30T06:25:37.903ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
36952025-04-30T06:25:37.903ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36962025-04-30T06:25:37.903ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36972025-04-30T06:25:37.903ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36982025-04-30T06:25:37.903ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36992025-04-30T06:25:37.904ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37002025-04-30T06:25:37.908ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37012025-04-30T06:25:37.913ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37022025-04-30T06:25:37.913ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37032025-04-30T06:25:37.913ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
37042025-04-30T06:25:37.913ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37052025-04-30T06:25:37.913ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37062025-04-30T06:25:37.916ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
37072025-04-30T06:25:37.917ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
37082025-04-30T06:25:37.917ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
37092025-04-30T06:25:37.917ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
37102025-04-30T06:25:37.934ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37112025-04-30T06:25:37.934ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
37122025-04-30T06:25:37.935ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37132025-04-30T06:25:37.961ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
37142025-04-30T06:25:37.961ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
37152025-04-30T06:25:37.962ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: aa 25 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .%..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58432 req_id = 8773e86b-c13b-49bf-8287-7da1b41be47b
37162025-04-30T06:25:37.963ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43557,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58432 req_id = 8773e86b-c13b-49bf-8287-7da1b41be47b
37172025-04-30T06:25:37.963ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
37182025-04-30T06:25:37.964ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
37192025-04-30T06:25:37.964ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
37202025-04-30T06:25:37.965ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58432 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8773e86b-c13b-49bf-8287-7da1b41be47b
37212025-04-30T06:25:37.967ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3202, tv_nsec: 766290307 } })
37222025-04-30T06:25:37.968ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
37232025-04-30T06:25:37.968ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
37242025-04-30T06:25:37.968ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: dc f2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62159 req_id = de5354a8-771b-4826-88f1-fe12500da23a
37252025-04-30T06:25:37.975ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56562,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62159 req_id = de5354a8-771b-4826-88f1-fe12500da23a
37262025-04-30T06:25:37.975ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
37272025-04-30T06:25:37.975ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62159 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = de5354a8-771b-4826-88f1-fe12500da23a
37282025-04-30T06:25:37.976ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 89 57 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .W..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56942 req_id = 53645386-b93b-4b9c-bb67-a1beb10a41ee
37292025-04-30T06:25:37.976ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35159,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56942 req_id = 53645386-b93b-4b9c-bb67-a1beb10a41ee
37302025-04-30T06:25:37.976ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
37312025-04-30T06:25:37.976ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56942 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 53645386-b93b-4b9c-bb67-a1beb10a41ee
37322025-04-30T06:25:38.066ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
37332025-04-30T06:25:38.077ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: dc 73 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .s..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62800 req_id = a580d1b2-a155-412b-a920-6c4f00c58b00
37342025-04-30T06:25:38.077ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56435,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62800 req_id = a580d1b2-a155-412b-a920-6c4f00c58b00
37352025-04-30T06:25:38.079ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
37362025-04-30T06:25:38.079ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56435, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62800 req_id = a580d1b2-a155-412b-a920-6c4f00c58b00
37372025-04-30T06:25:38.079ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
37382025-04-30T06:25:38.079ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
37392025-04-30T06:25:38.079ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
37402025-04-30T06:25:38.082ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "012c91d1-c94e-40f8-86c0-4779102befaa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37412025-04-30T06:25:38.082ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
37422025-04-30T06:25:38.086ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbc0861a-ac7c-4edb-8cac-8d61454ead27", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37432025-04-30T06:25:38.086ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
37442025-04-30T06:25:38.088ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c482f0d-bca1-4e6e-9485-150e338d664a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37452025-04-30T06:25:38.088ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
37462025-04-30T06:25:38.090ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "000882df-d19d-4569-b1d3-7258d3e67efc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37472025-04-30T06:25:38.090ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
37482025-04-30T06:25:38.092ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13d5961c-a6e2-4406-8ff0-e6bb375f1744", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37492025-04-30T06:25:38.092ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
37502025-04-30T06:25:38.110ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c11923af-8a39-4492-ac59-35fd712da316", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37512025-04-30T06:25:38.111ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
37522025-04-30T06:25:38.113ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d007c141-6454-4de6-903b-2aec87c4c06d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37532025-04-30T06:25:38.113ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
37542025-04-30T06:25:38.115ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9c3ea9d-4d58-4f94-881e-cfa6d1feddf7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37552025-04-30T06:25:38.115ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
37562025-04-30T06:25:38.116ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "283d83c4-415a-436c-b185-62267e55d6dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37572025-04-30T06:25:38.117ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
37582025-04-30T06:25:38.118ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7ce0efb-f8e7-484b-8e19-7095ecf89c58", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37592025-04-30T06:25:38.118ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
37602025-04-30T06:25:38.119ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f74eacaa-f8cc-4cda-981d-8b80950ef490", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37612025-04-30T06:25:38.119ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
37622025-04-30T06:25:38.120ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3b3d87f-9ba5-4cd2-bc3d-47f3650a6b80", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37632025-04-30T06:25:38.120ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
37642025-04-30T06:25:38.123ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef2df3c5-4472-4205-b33b-8b755c36f29c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37652025-04-30T06:25:38.123ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
37662025-04-30T06:25:38.126ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "979828cf-debe-4347-a223-832d90b76512", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37672025-04-30T06:25:38.126ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
37682025-04-30T06:25:38.129ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5f44a1d-1163-4f9b-b4f2-d46180d3e121", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37692025-04-30T06:25:38.129ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
37702025-04-30T06:25:38.132ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "529b86fd-b477-4019-bedc-941a364e114a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37712025-04-30T06:25:38.132ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
37722025-04-30T06:25:38.134ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "714cb710-88ef-49e2-afe4-86c52227b553", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37732025-04-30T06:25:38.134ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
37742025-04-30T06:25:38.137ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2380fbd0-b040-4a5d-8184-d923382c2452", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37752025-04-30T06:25:38.137ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
37762025-04-30T06:25:38.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "777e9f5d-b807-4d5c-9fd5-7824b2b4c543", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37772025-04-30T06:25:38.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
37782025-04-30T06:25:38.143ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0eb66577-6932-4e1c-8930-27a9da9cfb5f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37792025-04-30T06:25:38.143ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
37802025-04-30T06:25:38.150ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b9747ef-c775-4f3c-991a-cfc12f0a8cd4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37812025-04-30T06:25:38.151ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
37822025-04-30T06:25:38.163ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10e3832e-2cc0-48ad-ac39-a1d94d955aad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37832025-04-30T06:25:38.163ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
37842025-04-30T06:25:38.168ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
37852025-04-30T06:25:38.169ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f90fbf2-0146-400e-adda-be03ee5982ca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37862025-04-30T06:25:38.169ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
37872025-04-30T06:25:38.177ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97654462-7511-4c86-a2eb-5b1ae9ea7e7a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37882025-04-30T06:25:38.177ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
37892025-04-30T06:25:38.180ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16622171-1658-4bec-9650-d0dedd7e31b9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37902025-04-30T06:25:38.180ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
37912025-04-30T06:25:38.182ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45c2fb59-b789-45d3-a1b0-0cc3b621df8d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37922025-04-30T06:25:38.182ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
37932025-04-30T06:25:38.184ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e2591c6-2fac-4ff2-b319-2dc12f1c28c8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37942025-04-30T06:25:38.184ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
37952025-04-30T06:25:38.186ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be179714-312f-4a93-9691-0fb1c091e74c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37962025-04-30T06:25:38.186ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
37972025-04-30T06:25:38.189ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d4ff2a7-951c-40f8-82ff-9df3621805e2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
37982025-04-30T06:25:38.189ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
37992025-04-30T06:25:38.191ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45d3d2a6-6372-4e68-8b04-f21b0bb505a1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38002025-04-30T06:25:38.192ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
38012025-04-30T06:25:38.192ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8541ab1e-5c16-438f-a6fd-448b0133ac88", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38022025-04-30T06:25:38.192ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
38032025-04-30T06:25:38.193ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56d5b844-820d-492c-8ec2-12ab368cacec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38042025-04-30T06:25:38.194ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
38052025-04-30T06:25:38.195ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "631e744b-fdae-4555-9911-ec2ce14a5e5b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38062025-04-30T06:25:38.195ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
38072025-04-30T06:25:38.196ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a72cf92a-fbff-45e7-a21e-984e48824492", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38082025-04-30T06:25:38.196ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
38092025-04-30T06:25:38.198ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f5c4417-45f7-4a5d-b80a-d456827455ca", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38102025-04-30T06:25:38.198ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
38112025-04-30T06:25:38.200ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b800a2b3-2a2f-4082-a3fc-e62072b278af", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38122025-04-30T06:25:38.200ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
38132025-04-30T06:25:38.202ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87930479-213d-4e82-8ab7-c6bdfb217aa6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38142025-04-30T06:25:38.203ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
38152025-04-30T06:25:38.205ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8927c24-5c19-4d4c-922d-7e402df1373b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38162025-04-30T06:25:38.205ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
38172025-04-30T06:25:38.207ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7108749-c271-49c9-be70-704f55634429", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38182025-04-30T06:25:38.207ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
38192025-04-30T06:25:38.209ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "425d75eb-613b-4509-8de9-9ca1b17ed0a3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38202025-04-30T06:25:38.209ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
38212025-04-30T06:25:38.211ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d3f9c2-aab2-4a03-979e-110b5b4bf950", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38222025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
38232025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d847429-5e04-4423-805a-4ea306ae95c0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38242025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
38252025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb720660-8e13-4600-b0df-7557d1d03bf9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38262025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
38272025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac9e8cd-b436-46c5-aa8f-a8f360e3e5c4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38282025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
38292025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "313e3bc6-882b-4de2-a424-8b67c89e4c1d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38302025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
38312025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
38322025-04-30T06:25:38.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d3dc8c-77a4-4d18-b6c3-d41086fc7c5c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:37 GMT"} })
38332025-04-30T06:25:38.230ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
38342025-04-30T06:25:38.230ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38352025-04-30T06:25:38.230ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38362025-04-30T06:25:38.230ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38372025-04-30T06:25:38.230ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
38382025-04-30T06:25:38.230ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38392025-04-30T06:25:38.230ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c17c08a-aef9-414f-b510-3b6d24afc6f9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38402025-04-30T06:25:38.230ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
38412025-04-30T06:25:38.230ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38422025-04-30T06:25:38.237ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38432025-04-30T06:25:38.237ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38442025-04-30T06:25:38.237ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
38452025-04-30T06:25:38.237ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38462025-04-30T06:25:38.238ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7a0aaf4-6e49-4f7d-9aa2-b7ca7c56e73a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38472025-04-30T06:25:38.239ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
38482025-04-30T06:25:38.241ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38492025-04-30T06:25:38.241ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
38502025-04-30T06:25:38.241ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38512025-04-30T06:25:38.242ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0fd79f8-e14b-455e-9a70-8d5f2caa783d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38522025-04-30T06:25:38.242ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
38532025-04-30T06:25:38.244ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1940211e-6c39-49b8-b307-a3b000d7e9f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38542025-04-30T06:25:38.244ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38552025-04-30T06:25:38.244ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38562025-04-30T06:25:38.244ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38572025-04-30T06:25:38.244ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38582025-04-30T06:25:38.245ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38592025-04-30T06:25:38.251ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38602025-04-30T06:25:38.254ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38612025-04-30T06:25:38.254ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38622025-04-30T06:25:38.254ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38632025-04-30T06:25:38.254ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38642025-04-30T06:25:38.254ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38652025-04-30T06:25:38.255ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38662025-04-30T06:25:38.256ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38672025-04-30T06:25:38.260ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38682025-04-30T06:25:38.260ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38692025-04-30T06:25:38.260ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38702025-04-30T06:25:38.260ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38712025-04-30T06:25:38.302ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
38722025-04-30T06:25:38.302ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
38732025-04-30T06:25:38.302ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:63806 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
38742025-04-30T06:25:38.303ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:63806 req_id = bdd7a36c-a6e3-4d68-8616-6e64f766d761 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
38752025-04-30T06:25:38.303ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 460 local_addr = [::1]:42668 method = GET remote_addr = [::1]:63806 req_id = bdd7a36c-a6e3-4d68-8616-6e64f766d761 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
38762025-04-30T06:25:38.303ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdd7a36c-a6e3-4d68-8616-6e64f766d761", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38772025-04-30T06:25:38.343ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
38782025-04-30T06:25:38.343ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
38792025-04-30T06:25:38.344ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:33200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
38802025-04-30T06:25:38.344ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:33200 req_id = d32ffda9-3edc-4e84-b016-b86d27bbf848 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
38812025-04-30T06:25:38.345ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 182 local_addr = [::1]:47918 method = GET remote_addr = [::1]:33200 req_id = d32ffda9-3edc-4e84-b016-b86d27bbf848 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
38822025-04-30T06:25:38.345ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d32ffda9-3edc-4e84-b016-b86d27bbf848", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38832025-04-30T06:25:38.384ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
38842025-04-30T06:25:38.384ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
38852025-04-30T06:25:38.384ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:52792 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
38862025-04-30T06:25:38.385ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:52792 req_id = c2a41b69-85db-4e38-9cd8-8dfa53d515af sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
38872025-04-30T06:25:38.385ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 235 local_addr = [::1]:35744 method = GET remote_addr = [::1]:52792 req_id = c2a41b69-85db-4e38-9cd8-8dfa53d515af response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
38882025-04-30T06:25:38.385ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a41b69-85db-4e38-9cd8-8dfa53d515af", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38892025-04-30T06:25:38.430ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
38902025-04-30T06:25:38.430ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
38912025-04-30T06:25:38.431ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:62837 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38922025-04-30T06:25:38.432ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:62837 req_id = 958a79d5-ac7c-40c6-a058-6bf262c9fbbe sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38932025-04-30T06:25:38.433ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 454 local_addr = [::1]:35343 method = GET remote_addr = [::1]:62837 req_id = 958a79d5-ac7c-40c6-a058-6bf262c9fbbe response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38942025-04-30T06:25:38.433ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "958a79d5-ac7c-40c6-a058-6bf262c9fbbe", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
38952025-04-30T06:25:38.435ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38962025-04-30T06:25:38.435ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38972025-04-30T06:25:38.435ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
38982025-04-30T06:25:38.438ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38992025-04-30T06:25:38.438ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39002025-04-30T06:25:38.438ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39012025-04-30T06:25:38.438ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39022025-04-30T06:25:38.439ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39032025-04-30T06:25:38.446ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39042025-04-30T06:25:38.450ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
39052025-04-30T06:25:38.451ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39062025-04-30T06:25:38.451ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39072025-04-30T06:25:38.452ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39082025-04-30T06:25:38.476ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
39092025-04-30T06:25:38.527ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39102025-04-30T06:25:38.527ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39112025-04-30T06:25:38.528ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39122025-04-30T06:25:38.528ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
39132025-04-30T06:25:38.528ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39142025-04-30T06:25:38.530ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39152025-04-30T06:25:38.538ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39162025-04-30T06:25:38.538ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39172025-04-30T06:25:38.538ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
39182025-04-30T06:25:38.538ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39192025-04-30T06:25:38.596ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
39202025-04-30T06:25:38.596ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
39212025-04-30T06:25:38.597ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 693.341328ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
39222025-04-30T06:25:38.597ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
39232025-04-30T06:25:38.597ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
39242025-04-30T06:25:38.597ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39252025-04-30T06:25:38.597ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39262025-04-30T06:25:38.597ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39272025-04-30T06:25:38.597ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39282025-04-30T06:25:38.604ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39292025-04-30T06:25:38.605ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
39302025-04-30T06:25:38.605ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39312025-04-30T06:25:38.606ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39322025-04-30T06:25:38.610ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
39332025-04-30T06:25:38.610ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
39342025-04-30T06:25:38.610ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
39352025-04-30T06:25:38.611ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
39362025-04-30T06:25:38.614ZTRACtest_replacement_sanity (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39372025-04-30T06:25:38.614ZTRACtest_replacement_sanity (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
39382025-04-30T06:25:38.615ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
39392025-04-30T06:25:38.624ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
39402025-04-30T06:25:38.625ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cb c3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64468 req_id = 2a51db84-a6af-496b-8052-289081a86609
39412025-04-30T06:25:38.625ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52163,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64468 req_id = 2a51db84-a6af-496b-8052-289081a86609
39422025-04-30T06:25:38.626ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
39432025-04-30T06:25:38.626ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
39442025-04-30T06:25:38.626ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
39452025-04-30T06:25:38.626ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64468 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2a51db84-a6af-496b-8052-289081a86609
39462025-04-30T06:25:38.628ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3203, tv_nsec: 427038855 } })
39472025-04-30T06:25:38.628ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
39482025-04-30T06:25:38.628ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
39492025-04-30T06:25:38.628ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 97 6c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .l..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52960 req_id = 96dc4eca-9cdf-4e7a-852b-a68356f41a8c
39502025-04-30T06:25:38.629ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38764,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52960 req_id = 96dc4eca-9cdf-4e7a-852b-a68356f41a8c
39512025-04-30T06:25:38.629ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
39522025-04-30T06:25:38.629ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52960 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 96dc4eca-9cdf-4e7a-852b-a68356f41a8c
39532025-04-30T06:25:38.629ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 39 ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 9...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64549 req_id = 82f48bfa-1b56-42e0-999d-fa2e63ce3018
39542025-04-30T06:25:38.629ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14847,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64549 req_id = 82f48bfa-1b56-42e0-999d-fa2e63ce3018
39552025-04-30T06:25:38.630ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
39562025-04-30T06:25:38.630ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64549 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 82f48bfa-1b56-42e0-999d-fa2e63ce3018
39572025-04-30T06:25:38.757ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
39582025-04-30T06:25:38.758ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 84 ca 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52149 req_id = 3ce6c5e6-84bd-4999-ab9a-eaacc9fcdb08
39592025-04-30T06:25:38.758ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33994,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52149 req_id = 3ce6c5e6-84bd-4999-ab9a-eaacc9fcdb08
39602025-04-30T06:25:38.759ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
39612025-04-30T06:25:38.759ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 33994, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52149 req_id = 3ce6c5e6-84bd-4999-ab9a-eaacc9fcdb08
39622025-04-30T06:25:38.759ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
39632025-04-30T06:25:38.759ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
39642025-04-30T06:25:38.760ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
39652025-04-30T06:25:38.762ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a03562cf-6913-4aea-b619-ed38f084dbb0", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39662025-04-30T06:25:38.763ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
39672025-04-30T06:25:38.764ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "783cb380-bb68-4af9-93de-f248f370a34f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39682025-04-30T06:25:38.764ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
39692025-04-30T06:25:38.768ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "873d6a9a-47ce-4094-88af-14faa43371a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39702025-04-30T06:25:38.768ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
39712025-04-30T06:25:38.770ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d19ecbf-63f0-46ce-93e1-4e2aef1e83e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39722025-04-30T06:25:38.771ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
39732025-04-30T06:25:38.773ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f2f0853-00cd-4dce-a3f0-cf38e2749262", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39742025-04-30T06:25:38.774ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
39752025-04-30T06:25:38.776ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "289e130c-7bca-4c5f-9821-6c2a88004c6b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39762025-04-30T06:25:38.776ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
39772025-04-30T06:25:38.779ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "72d05a18-1048-4fdc-a596-058ae4423295", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39782025-04-30T06:25:38.780ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
39792025-04-30T06:25:38.791ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "244f4b85-e21a-45d4-b8cb-b90174c6b400", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39802025-04-30T06:25:38.791ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
39812025-04-30T06:25:38.792ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cea401b2-7b6d-409d-8e27-f8b4e1c0f966", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39822025-04-30T06:25:38.792ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
39832025-04-30T06:25:38.794ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16d53a90-9961-4645-8044-62e27877f39c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39842025-04-30T06:25:38.794ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
39852025-04-30T06:25:38.795ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb4ba7f7-ef93-436b-8dda-f0ced8ffafc7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39862025-04-30T06:25:38.795ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
39872025-04-30T06:25:38.798ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acfaf20f-f089-40ef-84a1-834b3386d2c5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39882025-04-30T06:25:38.798ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
39892025-04-30T06:25:38.800ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5eeddf3f-1c36-408a-9e45-5bfbd621c60e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39902025-04-30T06:25:38.800ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
39912025-04-30T06:25:38.804ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec0f523-c262-459a-a3d8-af2d9cf27aff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39922025-04-30T06:25:38.804ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
39932025-04-30T06:25:38.807ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97fb9e6c-51c3-4b7c-85f8-6d0c52b284f6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39942025-04-30T06:25:38.807ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
39952025-04-30T06:25:38.809ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9868a99-3749-4cde-8bf7-e632ffb1cbb7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39962025-04-30T06:25:38.809ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
39972025-04-30T06:25:38.811ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9e81a8b-2da0-4919-a38c-10b3867a22b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
39982025-04-30T06:25:38.812ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
39992025-04-30T06:25:38.814ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb30e02a-aea4-46f4-9a53-92570482d073", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40002025-04-30T06:25:38.814ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
40012025-04-30T06:25:38.816ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14aa3e50-44b2-4cf1-999f-b3b8c32ae854", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40022025-04-30T06:25:38.816ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
40032025-04-30T06:25:38.817ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c109c9c-b003-41da-8f62-10a0c53eeaed", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40042025-04-30T06:25:38.817ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
40052025-04-30T06:25:38.819ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1310d416-9883-41f7-8352-c61516f8d8a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40062025-04-30T06:25:38.819ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
40072025-04-30T06:25:38.820ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57e1e845-3158-464a-bafd-0ac4323d4f64", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40082025-04-30T06:25:38.820ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
40092025-04-30T06:25:38.822ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b50a5c8-05e3-42ef-9575-9de0e5f7c205", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40102025-04-30T06:25:38.822ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
40112025-04-30T06:25:38.824ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e110e0-c801-4784-9300-fe56f0fa8f1c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40122025-04-30T06:25:38.825ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
40132025-04-30T06:25:38.827ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2c6da9d-ce00-4702-bc01-46c360af0e33", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40142025-04-30T06:25:38.827ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
40152025-04-30T06:25:38.830ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e6bc837-e5dc-4b62-80f6-46f3f375aa52", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40162025-04-30T06:25:38.830ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
40172025-04-30T06:25:38.874ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
40182025-04-30T06:25:38.881ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "165a869b-6429-4523-80a5-fb26dd993769", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40192025-04-30T06:25:38.881ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
40202025-04-30T06:25:38.886ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3c7a02c-1e3f-49ff-badd-203c96a1785f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40212025-04-30T06:25:38.886ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
40222025-04-30T06:25:38.892ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b1676d4-d5a9-4755-884f-29adce189c1a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40232025-04-30T06:25:38.892ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
40242025-04-30T06:25:38.895ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cd22a9a-8ece-4a57-9986-41df1ed12164", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40252025-04-30T06:25:38.895ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
40262025-04-30T06:25:38.898ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75c9c018-ea11-4359-99d8-e7f8da9b70a6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40272025-04-30T06:25:38.899ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
40282025-04-30T06:25:38.904ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "690f9cd5-6571-4422-ad1f-9d960f01d6a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40292025-04-30T06:25:38.904ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
40302025-04-30T06:25:38.915ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c07e83b-38e5-4a67-9c3f-ca06c7211a30", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40312025-04-30T06:25:38.915ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
40322025-04-30T06:25:38.917ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "844d15c5-16d2-4361-81fe-5fa56632e9d8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40332025-04-30T06:25:38.917ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
40342025-04-30T06:25:38.919ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d3bf39a-a7ca-4257-ba2c-69b56891d92e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40352025-04-30T06:25:38.919ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
40362025-04-30T06:25:38.922ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82e66277-c03c-436b-a7fe-93596aa1d7e1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40372025-04-30T06:25:38.922ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
40382025-04-30T06:25:38.925ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40392025-04-30T06:25:38.926ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40402025-04-30T06:25:38.926ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40412025-04-30T06:25:38.926ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
40422025-04-30T06:25:38.927ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40432025-04-30T06:25:38.929ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "516aea8f-1b2f-445c-a0ad-893240514cf5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40442025-04-30T06:25:38.929ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
40452025-04-30T06:25:38.931ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40462025-04-30T06:25:38.939ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40472025-04-30T06:25:38.941ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40482025-04-30T06:25:38.941ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
40492025-04-30T06:25:38.941ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40502025-04-30T06:25:38.942ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40512025-04-30T06:25:38.942ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
40522025-04-30T06:25:38.943ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40532025-04-30T06:25:38.943ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e33a53c-d218-47b3-a0a4-c233bc3c086c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40542025-04-30T06:25:38.944ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
40552025-04-30T06:25:38.947ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ba5eef0-71e1-4066-a086-091a518df6d3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40562025-04-30T06:25:38.947ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
40572025-04-30T06:25:38.950ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d9a749e-9db3-44b7-97e6-0600809499c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40582025-04-30T06:25:38.951ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
40592025-04-30T06:25:38.953ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e65bea39-306b-4961-b1fb-80b391ad7a6e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40602025-04-30T06:25:38.954ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
40612025-04-30T06:25:38.955ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b604ee8-3d6c-44bd-b1e0-cee55bf432f4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40622025-04-30T06:25:38.955ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
40632025-04-30T06:25:38.957ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6fb105a-7a4f-427f-986b-d3c859db2340", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40642025-04-30T06:25:38.957ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
40652025-04-30T06:25:38.971ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05ab228e-855e-49b5-8d9a-6143a9a3279f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40662025-04-30T06:25:38.971ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
40672025-04-30T06:25:38.972ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40c3f27-68ab-48f0-be7f-f018e48bd37a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40682025-04-30T06:25:38.972ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
40692025-04-30T06:25:38.972ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
40702025-04-30T06:25:38.977ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "54cc616c-e13e-4da6-826b-2b7f480d3a62", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40712025-04-30T06:25:38.977ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
40722025-04-30T06:25:38.979ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c06ef46f-c133-4469-943d-bbd85bf36a19", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40732025-04-30T06:25:38.979ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
40742025-04-30T06:25:38.982ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5297916-beb1-4a51-8a3d-a6d3e97d6c5c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40752025-04-30T06:25:38.982ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
40762025-04-30T06:25:38.984ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3260f4b1-4148-44be-a19d-a9c73484e12d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40772025-04-30T06:25:38.985ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
40782025-04-30T06:25:38.988ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
40792025-04-30T06:25:38.989ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1939f0e5-867b-4811-bfab-16a3dbafc7aa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
40802025-04-30T06:25:38.990ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40812025-04-30T06:25:38.990ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40822025-04-30T06:25:38.990ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40832025-04-30T06:25:38.990ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40842025-04-30T06:25:38.991ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40852025-04-30T06:25:38.997ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40862025-04-30T06:25:39.004ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40872025-04-30T06:25:39.004ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40882025-04-30T06:25:39.005ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40892025-04-30T06:25:39.005ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40902025-04-30T06:25:39.005ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40912025-04-30T06:25:39.006ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40922025-04-30T06:25:39.010ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40932025-04-30T06:25:39.017ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40942025-04-30T06:25:39.017ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40952025-04-30T06:25:39.017ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40962025-04-30T06:25:39.018ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40972025-04-30T06:25:39.040ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40982025-04-30T06:25:39.040ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40992025-04-30T06:25:39.040ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41002025-04-30T06:25:39.040ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
41012025-04-30T06:25:39.040ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41022025-04-30T06:25:39.099ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
41032025-04-30T06:25:39.099ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
41042025-04-30T06:25:39.100ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:35561 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
41052025-04-30T06:25:39.103ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41062025-04-30T06:25:39.120ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41072025-04-30T06:25:39.120ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41082025-04-30T06:25:39.120ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
41092025-04-30T06:25:39.120ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41102025-04-30T06:25:39.121ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:35561 req_id = 88bc9e7e-eb54-4fab-8b38-8118e00b6198 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
41112025-04-30T06:25:39.121ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 617 local_addr = [::1]:42668 method = GET remote_addr = [::1]:35561 req_id = 88bc9e7e-eb54-4fab-8b38-8118e00b6198 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
41122025-04-30T06:25:39.122ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "88bc9e7e-eb54-4fab-8b38-8118e00b6198", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
41132025-04-30T06:25:39.186ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
41142025-04-30T06:25:39.186ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
41152025-04-30T06:25:39.186ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:33093 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
41162025-04-30T06:25:39.188ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:33093 req_id = 2a380637-97da-41b9-bf34-8f0e2dd0f80d sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
41172025-04-30T06:25:39.189ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1150 local_addr = [::1]:47918 method = GET remote_addr = [::1]:33093 req_id = 2a380637-97da-41b9-bf34-8f0e2dd0f80d response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
41182025-04-30T06:25:39.189ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41192025-04-30T06:25:39.189ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
41202025-04-30T06:25:39.190ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41212025-04-30T06:25:39.191ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a380637-97da-41b9-bf34-8f0e2dd0f80d", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:38 GMT"} })
41222025-04-30T06:25:39.256ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
41232025-04-30T06:25:39.257ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
41242025-04-30T06:25:39.257ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:55967 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
41252025-04-30T06:25:39.259ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:55967 req_id = b4a83a99-d332-40c3-b2f9-34876bd6dd1d sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
41262025-04-30T06:25:39.268ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 9469 local_addr = [::1]:35744 method = GET remote_addr = [::1]:55967 req_id = b4a83a99-d332-40c3-b2f9-34876bd6dd1d response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
41272025-04-30T06:25:39.271ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4a83a99-d332-40c3-b2f9-34876bd6dd1d", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
41282025-04-30T06:25:39.344ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
41292025-04-30T06:25:39.345ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
41302025-04-30T06:25:39.346ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:52003 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
41312025-04-30T06:25:39.346ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
41322025-04-30T06:25:39.352ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41332025-04-30T06:25:39.353ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 0f52a782-495c-421d-a727-ba6f5b33b615 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41342025-04-30T06:25:39.353ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:52003 req_id = ad6e0e52-dddf-4482-99af-9db072aa2dd2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
41352025-04-30T06:25:39.354ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 0f52a782-495c-421d-a727-ba6f5b33b615 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41362025-04-30T06:25:39.354ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 0f52a782-495c-421d-a727-ba6f5b33b615 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41372025-04-30T06:25:39.355ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 0f52a782-495c-421d-a727-ba6f5b33b615 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41382025-04-30T06:25:39.355ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1856 local_addr = [::1]:35343 method = GET remote_addr = [::1]:52003 req_id = ad6e0e52-dddf-4482-99af-9db072aa2dd2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
41392025-04-30T06:25:39.362ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad6e0e52-dddf-4482-99af-9db072aa2dd2", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
41402025-04-30T06:25:39.366ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
41412025-04-30T06:25:39.366ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
41422025-04-30T06:25:39.366ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
41432025-04-30T06:25:39.372ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
41442025-04-30T06:25:39.372ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41452025-04-30T06:25:39.372ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41462025-04-30T06:25:39.373ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41472025-04-30T06:25:39.373ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41482025-04-30T06:25:39.377ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 23907 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 0f52a782-495c-421d-a727-ba6f5b33b615 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
41492025-04-30T06:25:39.380ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f52a782-495c-421d-a727-ba6f5b33b615", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
41502025-04-30T06:25:39.380ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
41512025-04-30T06:25:39.381ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41522025-04-30T06:25:39.387ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
41532025-04-30T06:25:39.388ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41542025-04-30T06:25:39.388ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41552025-04-30T06:25:39.388ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41562025-04-30T06:25:39.390ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = ad9afeb3-e808-4642-97b5-f8c9f9e55fd9 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
41572025-04-30T06:25:39.391ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = ad9afeb3-e808-4642-97b5-f8c9f9e55fd9 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
41582025-04-30T06:25:39.391ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = ad9afeb3-e808-4642-97b5-f8c9f9e55fd9 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
41592025-04-30T06:25:39.392ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = ad9afeb3-e808-4642-97b5-f8c9f9e55fd9 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
41602025-04-30T06:25:39.403ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 12037 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = ad9afeb3-e808-4642-97b5-f8c9f9e55fd9 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
41612025-04-30T06:25:39.406ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad9afeb3-e808-4642-97b5-f8c9f9e55fd9", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
41622025-04-30T06:25:39.406ZDEBGtest_replacement_sanity (oximeter-agent): registered new metric producer address = [::1]:56181 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41632025-04-30T06:25:39.490ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
41642025-04-30T06:25:39.494ZDEBGtest_replacement_sanity (oximeter-agent): registered new metric producer address = 127.0.0.1:50502 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
41652025-04-30T06:25:39.568ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
41662025-04-30T06:25:39.575ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
41672025-04-30T06:25:39.590ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
41682025-04-30T06:25:39.608ZTRACtest_replacement_sanity (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41692025-04-30T06:25:39.608ZTRACtest_replacement_sanity (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
41702025-04-30T06:25:39.641ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41712025-04-30T06:25:39.641ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41722025-04-30T06:25:39.641ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41732025-04-30T06:25:39.641ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
41742025-04-30T06:25:39.643ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41752025-04-30T06:25:39.655ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41762025-04-30T06:25:39.662ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41772025-04-30T06:25:39.662ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41782025-04-30T06:25:39.662ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
41792025-04-30T06:25:39.663ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41802025-04-30T06:25:39.688ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
41812025-04-30T06:25:39.689ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
41822025-04-30T06:25:39.689ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.0922917s iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
41832025-04-30T06:25:39.689ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
41842025-04-30T06:25:39.689ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
41852025-04-30T06:25:39.689ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41862025-04-30T06:25:39.689ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41872025-04-30T06:25:39.689ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41882025-04-30T06:25:39.692ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41892025-04-30T06:25:39.692ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41902025-04-30T06:25:39.692ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
41912025-04-30T06:25:39.693ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41922025-04-30T06:25:39.695ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41932025-04-30T06:25:39.708ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
41942025-04-30T06:25:39.709ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
41952025-04-30T06:25:39.709ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
41962025-04-30T06:25:39.709ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
41972025-04-30T06:25:39.761ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
41982025-04-30T06:25:39.761ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
41992025-04-30T06:25:39.762ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1c 40 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .@..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54800 req_id = 4c922919-0c27-48e9-868e-a6ca8a114b9a
42002025-04-30T06:25:39.763ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7232,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54800 req_id = 4c922919-0c27-48e9-868e-a6ca8a114b9a
42012025-04-30T06:25:39.763ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
42022025-04-30T06:25:39.763ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
42032025-04-30T06:25:39.764ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42042025-04-30T06:25:39.765ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54800 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4c922919-0c27-48e9-868e-a6ca8a114b9a
42052025-04-30T06:25:39.766ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3204, tv_nsec: 564794740 } })
42062025-04-30T06:25:39.766ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
42072025-04-30T06:25:39.766ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
42082025-04-30T06:25:39.766ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1f cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52589 req_id = 09196c22-54fd-4032-b818-af075982c343
42092025-04-30T06:25:39.767ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8139,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52589 req_id = 09196c22-54fd-4032-b818-af075982c343
42102025-04-30T06:25:39.767ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
42112025-04-30T06:25:39.767ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52589 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 09196c22-54fd-4032-b818-af075982c343
42122025-04-30T06:25:39.767ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 10 c3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49284 req_id = 6fb9f64b-289d-430b-ac83-8ccd9b0b2ebb
42132025-04-30T06:25:39.767ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4291,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49284 req_id = 6fb9f64b-289d-430b-ac83-8ccd9b0b2ebb
42142025-04-30T06:25:39.769ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42152025-04-30T06:25:39.769ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49284 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6fb9f64b-289d-430b-ac83-8ccd9b0b2ebb
42162025-04-30T06:25:39.862ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42172025-04-30T06:25:39.863ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 56 13 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c V............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62760 req_id = ab0e6884-f2a2-450f-8e3b-eaf9ec9b77f2
42182025-04-30T06:25:39.863ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22035,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62760 req_id = ab0e6884-f2a2-450f-8e3b-eaf9ec9b77f2
42192025-04-30T06:25:39.864ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
42202025-04-30T06:25:39.865ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22035, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62760 req_id = ab0e6884-f2a2-450f-8e3b-eaf9ec9b77f2
42212025-04-30T06:25:39.865ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
42222025-04-30T06:25:39.865ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
42232025-04-30T06:25:39.865ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
42242025-04-30T06:25:39.867ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbbe6107-0953-4b52-9bdc-01031a8784a3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42252025-04-30T06:25:39.868ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
42262025-04-30T06:25:39.870ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d505b3-7237-45df-ae02-c78a7f878b57", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42272025-04-30T06:25:39.870ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
42282025-04-30T06:25:39.872ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07e2cc69-d843-4422-a656-8e28b52648da", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42292025-04-30T06:25:39.873ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
42302025-04-30T06:25:39.876ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f691715-b398-4dcc-8cf1-87e66d63c56e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42312025-04-30T06:25:39.876ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
42322025-04-30T06:25:39.879ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a01b7b2f-602a-4cec-99e9-334ede4b4589", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42332025-04-30T06:25:39.879ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
42342025-04-30T06:25:39.882ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f65fee40-313a-4300-9b00-4c4a2a559259", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42352025-04-30T06:25:39.882ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
42362025-04-30T06:25:39.886ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e61e1548-cbf0-4b57-b2cf-a64b48348fc7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42372025-04-30T06:25:39.886ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
42382025-04-30T06:25:39.886ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
42392025-04-30T06:25:39.900ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b02fdef6-9230-44db-9212-2c76ba3825ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42402025-04-30T06:25:39.900ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
42412025-04-30T06:25:39.901ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3c7fde-e222-41fe-89fc-d8d70b0fc4ff", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42422025-04-30T06:25:39.902ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
42432025-04-30T06:25:39.907ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "caba28ca-e1c7-45e2-b872-c94d643b0fba", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42442025-04-30T06:25:39.908ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
42452025-04-30T06:25:39.909ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d4a1c71-2363-4f80-874b-21462a713221", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42462025-04-30T06:25:39.909ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
42472025-04-30T06:25:39.910ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e792cd4f-9fa0-454e-a20d-a24610202b17", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42482025-04-30T06:25:39.910ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
42492025-04-30T06:25:39.912ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b79bd49-de29-4651-938b-19dcee2d84f6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42502025-04-30T06:25:39.912ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
42512025-04-30T06:25:39.919ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b866a89-725a-44bb-a97e-f4bc85c30dcf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42522025-04-30T06:25:39.919ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
42532025-04-30T06:25:39.922ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cae3f69a-9c97-4476-8ea2-e7158c36bed6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42542025-04-30T06:25:39.922ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
42552025-04-30T06:25:39.926ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dc5b7df-6e80-4d84-bab3-d8bc2eea5afd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42562025-04-30T06:25:39.926ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
42572025-04-30T06:25:39.928ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b136725-23ec-4c92-a7f1-70a616fd1e80", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42582025-04-30T06:25:39.928ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
42592025-04-30T06:25:39.931ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c000ba2c-4b15-4517-85f2-74a0c2d98776", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42602025-04-30T06:25:39.931ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
42612025-04-30T06:25:39.933ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e9818f9-1556-45b0-8745-c2c004506406", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42622025-04-30T06:25:39.933ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
42632025-04-30T06:25:39.935ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a4bc60e-9bf6-4050-9474-615f47614f33", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42642025-04-30T06:25:39.935ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
42652025-04-30T06:25:39.936ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b99707c4-614d-4f43-a6a1-69220953d9b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42662025-04-30T06:25:39.936ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
42672025-04-30T06:25:39.937ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2174ddb-dff4-4626-b3ea-f1da5d54517e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42682025-04-30T06:25:39.937ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
42692025-04-30T06:25:39.938ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42702025-04-30T06:25:39.938ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42712025-04-30T06:25:39.938ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42722025-04-30T06:25:39.938ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
42732025-04-30T06:25:39.938ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42742025-04-30T06:25:39.940ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa5f1c86-6143-4cd9-80e4-9ea59c80a5f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42752025-04-30T06:25:39.940ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
42762025-04-30T06:25:39.941ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9060c6cb-9b9c-4c16-8463-09bab6d4ed9f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42772025-04-30T06:25:39.941ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
42782025-04-30T06:25:39.942ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42792025-04-30T06:25:39.947ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42802025-04-30T06:25:39.947ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42812025-04-30T06:25:39.947ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
42822025-04-30T06:25:39.947ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42832025-04-30T06:25:39.951ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42842025-04-30T06:25:39.951ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
42852025-04-30T06:25:39.951ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42862025-04-30T06:25:39.953ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ff1e273-806c-41d1-9024-ac61f1a7fbbc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42872025-04-30T06:25:39.953ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
42882025-04-30T06:25:39.957ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b11813-88e8-407e-8c9c-354232153237", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42892025-04-30T06:25:39.957ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
42902025-04-30T06:25:39.961ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73df92ea-aec2-47d9-96a8-f2d1dbafa6d9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42912025-04-30T06:25:39.961ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
42922025-04-30T06:25:39.969ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f191d5-f610-4bcc-aa82-a5e7bed29866", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42932025-04-30T06:25:39.969ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
42942025-04-30T06:25:39.974ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e43c3a64-1ded-45a6-9d59-d12265d47cba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42952025-04-30T06:25:39.974ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
42962025-04-30T06:25:39.977ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e209e86-759f-4902-8725-7531a20e6f63", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42972025-04-30T06:25:39.978ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
42982025-04-30T06:25:39.983ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28767df1-0bd7-4909-984d-2734ba89a27e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
42992025-04-30T06:25:39.984ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
43002025-04-30T06:25:39.985ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
43012025-04-30T06:25:39.986ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f31d8a21-fa31-412f-9498-5d5256d22c12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43022025-04-30T06:25:39.987ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
43032025-04-30T06:25:39.988ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4cf1b5e-5e7b-445f-a8b0-2a6b1fd7d844", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43042025-04-30T06:25:39.988ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
43052025-04-30T06:25:39.989ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2bc13c6-5c38-4ce2-8d56-54fcd342658c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43062025-04-30T06:25:39.989ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
43072025-04-30T06:25:39.990ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8b3d9fb-81c4-4714-9222-3e1b48e37f0b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43082025-04-30T06:25:39.990ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
43092025-04-30T06:25:39.992ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28de00ac-2bde-4692-8932-52e95dc34920", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43102025-04-30T06:25:39.993ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
43112025-04-30T06:25:39.996ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25870bea-9089-4324-8829-8a4d572485fc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43122025-04-30T06:25:39.996ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
43132025-04-30T06:25:39.998ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b228f709-839e-41af-9d57-31116f27d4b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43142025-04-30T06:25:39.998ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
43152025-04-30T06:25:40.004ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70c892cd-06f9-4ce1-a680-72b0bb08453e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43162025-04-30T06:25:40.005ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
43172025-04-30T06:25:40.013ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be4c375b-2045-41c7-bbf9-3c413fc50fef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43182025-04-30T06:25:40.016ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
43192025-04-30T06:25:40.018ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b92aae41-87f4-4ae6-82c5-2a9163d2a3a4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43202025-04-30T06:25:40.019ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
43212025-04-30T06:25:40.019ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d499346f-88b0-45e6-a25b-c505efac4765", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43222025-04-30T06:25:40.020ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
43232025-04-30T06:25:40.021ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b1901ed-2864-4a67-b634-2e844b6818cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43242025-04-30T06:25:40.021ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
43252025-04-30T06:25:40.022ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e283bfec-fdf0-4b74-bd2e-121b71ee3197", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43262025-04-30T06:25:40.022ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
43272025-04-30T06:25:40.023ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e826d47c-8ba8-48a8-8268-7803e9652e4c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43282025-04-30T06:25:40.023ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
43292025-04-30T06:25:40.023ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
43302025-04-30T06:25:40.025ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a177d541-cc65-40a1-8c04-c636e507a52c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43312025-04-30T06:25:40.025ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
43322025-04-30T06:25:40.027ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7b98893-13a9-466d-bb80-4086dd682b14", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43332025-04-30T06:25:40.027ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
43342025-04-30T06:25:40.028ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06787d0e-8b09-4d0e-bb54-41d081bc31df", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43352025-04-30T06:25:40.029ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
43362025-04-30T06:25:40.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ada5eedd-8548-494f-967b-945b6a3e8e97", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43372025-04-30T06:25:40.031ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
43382025-04-30T06:25:40.032ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dccca85c-7d1e-4daa-ba1f-343cf5d8c894", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43392025-04-30T06:25:40.032ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
43402025-04-30T06:25:40.032ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43412025-04-30T06:25:40.032ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43422025-04-30T06:25:40.032ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43432025-04-30T06:25:40.033ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43442025-04-30T06:25:40.037ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43452025-04-30T06:25:40.037ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43462025-04-30T06:25:40.037ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43472025-04-30T06:25:40.037ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
43482025-04-30T06:25:40.038ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43492025-04-30T06:25:40.039ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43502025-04-30T06:25:40.047ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
43512025-04-30T06:25:40.047ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
43522025-04-30T06:25:40.047ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43532025-04-30T06:25:40.047ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43542025-04-30T06:25:40.047ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43552025-04-30T06:25:40.047ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43562025-04-30T06:25:40.049ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43572025-04-30T06:25:40.057ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43582025-04-30T06:25:40.057ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43592025-04-30T06:25:40.057ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
43602025-04-30T06:25:40.059ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43612025-04-30T06:25:40.059ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43622025-04-30T06:25:40.065ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
43632025-04-30T06:25:40.065ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
43642025-04-30T06:25:40.065ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
43652025-04-30T06:25:40.066ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
43662025-04-30T06:25:40.070ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43672025-04-30T06:25:40.070ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
43682025-04-30T06:25:40.071ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43692025-04-30T06:25:40.132ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
43702025-04-30T06:25:40.132ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
43712025-04-30T06:25:40.134ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:54522 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
43722025-04-30T06:25:40.135ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:54522 req_id = 2e7188c4-00f0-417d-b357-a1792d948831 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
43732025-04-30T06:25:40.136ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 464 local_addr = [::1]:42668 method = GET remote_addr = [::1]:54522 req_id = 2e7188c4-00f0-417d-b357-a1792d948831 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
43742025-04-30T06:25:40.136ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e7188c4-00f0-417d-b357-a1792d948831", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43752025-04-30T06:25:40.199ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
43762025-04-30T06:25:40.199ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
43772025-04-30T06:25:40.200ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:52393 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
43782025-04-30T06:25:40.203ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:52393 req_id = a4355d1b-c9f1-4f88-ac30-2fb66839d973 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
43792025-04-30T06:25:40.203ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 380 local_addr = [::1]:47918 method = GET remote_addr = [::1]:52393 req_id = a4355d1b-c9f1-4f88-ac30-2fb66839d973 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
43802025-04-30T06:25:40.204ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4355d1b-c9f1-4f88-ac30-2fb66839d973", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:39 GMT"} })
43812025-04-30T06:25:40.297ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
43822025-04-30T06:25:40.297ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
43832025-04-30T06:25:40.300ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:60933 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
43842025-04-30T06:25:40.301ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:60933 req_id = 13e401ea-3a18-4d69-90ec-941c6e130030 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
43852025-04-30T06:25:40.308ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 6980 local_addr = [::1]:35744 method = GET remote_addr = [::1]:60933 req_id = 13e401ea-3a18-4d69-90ec-941c6e130030 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
43862025-04-30T06:25:40.310ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "13e401ea-3a18-4d69-90ec-941c6e130030", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
43872025-04-30T06:25:40.401ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
43882025-04-30T06:25:40.402ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
43892025-04-30T06:25:40.404ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:49820 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
43902025-04-30T06:25:40.410ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:49820 req_id = 26dfde5e-981a-4eb5-8a87-a7854b9b374b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
43912025-04-30T06:25:40.410ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43922025-04-30T06:25:40.411ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43932025-04-30T06:25:40.411ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 957 local_addr = [::1]:35343 method = GET remote_addr = [::1]:49820 req_id = 26dfde5e-981a-4eb5-8a87-a7854b9b374b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
43942025-04-30T06:25:40.419ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "26dfde5e-981a-4eb5-8a87-a7854b9b374b", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
43952025-04-30T06:25:40.422ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
43962025-04-30T06:25:40.422ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
43972025-04-30T06:25:40.422ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
43982025-04-30T06:25:40.429ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
43992025-04-30T06:25:40.429ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44002025-04-30T06:25:40.429ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44012025-04-30T06:25:40.429ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44022025-04-30T06:25:40.430ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44032025-04-30T06:25:40.439ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44042025-04-30T06:25:40.444ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44052025-04-30T06:25:40.444ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44062025-04-30T06:25:40.444ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44072025-04-30T06:25:40.445ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44082025-04-30T06:25:40.449ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44092025-04-30T06:25:40.451ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44102025-04-30T06:25:40.451ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44112025-04-30T06:25:40.451ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
44122025-04-30T06:25:40.480ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
44132025-04-30T06:25:40.517ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
44142025-04-30T06:25:40.518ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
44152025-04-30T06:25:40.518ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 829.24185ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
44162025-04-30T06:25:40.518ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
44172025-04-30T06:25:40.518ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44182025-04-30T06:25:40.519ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44192025-04-30T06:25:40.519ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44202025-04-30T06:25:40.519ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44212025-04-30T06:25:40.520ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44222025-04-30T06:25:40.523ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44232025-04-30T06:25:40.528ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44242025-04-30T06:25:40.528ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44252025-04-30T06:25:40.528ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44262025-04-30T06:25:40.528ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44272025-04-30T06:25:40.533ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44282025-04-30T06:25:40.533ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44292025-04-30T06:25:40.533ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44302025-04-30T06:25:40.533ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
44312025-04-30T06:25:40.534ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44322025-04-30T06:25:40.535ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
44332025-04-30T06:25:40.535ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
44342025-04-30T06:25:40.537ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cf 27 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .'..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53320 req_id = 8f81d3e7-cda9-4dd9-a63e-9f198c25b777
44352025-04-30T06:25:40.537ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53031,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53320 req_id = 8f81d3e7-cda9-4dd9-a63e-9f198c25b777
44362025-04-30T06:25:40.538ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
44372025-04-30T06:25:40.538ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
44382025-04-30T06:25:40.539ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44392025-04-30T06:25:40.539ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53320 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8f81d3e7-cda9-4dd9-a63e-9f198c25b777
44402025-04-30T06:25:40.539ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44412025-04-30T06:25:40.548ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44422025-04-30T06:25:40.549ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44432025-04-30T06:25:40.549ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
44442025-04-30T06:25:40.550ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44452025-04-30T06:25:40.551ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3205, tv_nsec: 350415628 } })
44462025-04-30T06:25:40.551ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
44472025-04-30T06:25:40.553ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
44482025-04-30T06:25:40.554ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c0 e3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58983 req_id = 0ccde7de-ccea-4cb2-bc98-4d6288ef5031
44492025-04-30T06:25:40.554ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49379,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58983 req_id = 0ccde7de-ccea-4cb2-bc98-4d6288ef5031
44502025-04-30T06:25:40.554ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
44512025-04-30T06:25:40.554ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58983 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0ccde7de-ccea-4cb2-bc98-4d6288ef5031
44522025-04-30T06:25:40.554ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 36 36 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 66..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56947 req_id = d58e4379-a149-4fff-8e7a-2aaf1bac4fb9
44532025-04-30T06:25:40.554ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13878,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56947 req_id = d58e4379-a149-4fff-8e7a-2aaf1bac4fb9
44542025-04-30T06:25:40.555ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
44552025-04-30T06:25:40.555ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56947 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d58e4379-a149-4fff-8e7a-2aaf1bac4fb9
44562025-04-30T06:25:40.657ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
44572025-04-30T06:25:40.658ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44582025-04-30T06:25:40.658ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
44592025-04-30T06:25:40.659ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44602025-04-30T06:25:40.659ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44612025-04-30T06:25:40.659ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44622025-04-30T06:25:40.659ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
44632025-04-30T06:25:40.660ZDEBGtest_replacement_sanity (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
44642025-04-30T06:25:40.660ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
44652025-04-30T06:25:40.676ZTRACtest_replacement_sanity (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
44662025-04-30T06:25:40.676ZDEBGtest_replacement_sanity (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:25:40.660512601Z } timeseries_name = integration_target:integration_metric
44672025-04-30T06:25:40.677ZDEBGtest_replacement_sanity (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
44682025-04-30T06:25:40.677ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
44692025-04-30T06:25:40.677ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5e 10 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ^............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64737 req_id = 597f0b29-c962-4b2d-af46-b1d3ebd9ce49
44702025-04-30T06:25:40.677ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24080,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64737 req_id = 597f0b29-c962-4b2d-af46-b1d3ebd9ce49
44712025-04-30T06:25:40.678ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
44722025-04-30T06:25:40.678ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24080, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64737 req_id = 597f0b29-c962-4b2d-af46-b1d3ebd9ce49
44732025-04-30T06:25:40.681ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
44742025-04-30T06:25:40.682ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
44752025-04-30T06:25:40.682ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
44762025-04-30T06:25:40.686ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
44772025-04-30T06:25:40.688ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "47e9a1e8-f565-40bb-8c7f-7bdf6344067d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44782025-04-30T06:25:40.688ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
44792025-04-30T06:25:40.691ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
44802025-04-30T06:25:40.691ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
44812025-04-30T06:25:40.691ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b41c3c6f-67ae-4466-96f5-c2c98cc20a11", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44822025-04-30T06:25:40.691ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
44832025-04-30T06:25:40.694ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
44842025-04-30T06:25:40.694ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
44852025-04-30T06:25:40.695ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ac2350-f91d-4539-8150-82955c07c3a7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44862025-04-30T06:25:40.695ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
44872025-04-30T06:25:40.701ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d000318-162e-489f-a1f4-99d5b442ffb7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44882025-04-30T06:25:40.702ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
44892025-04-30T06:25:40.724ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ac27106-96af-4c35-b1a0-c7761dc65148", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44902025-04-30T06:25:40.725ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
44912025-04-30T06:25:40.725ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
44922025-04-30T06:25:40.730ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a4d8b2b-18fb-4134-8dba-a79448120bfc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44932025-04-30T06:25:40.730ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
44942025-04-30T06:25:40.733ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d215e08-f8b1-4fb6-ac5d-0b597a980f9d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44952025-04-30T06:25:40.733ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
44962025-04-30T06:25:40.735ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "287b139d-63d8-41e0-871f-c61ac9b9d5ae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44972025-04-30T06:25:40.735ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
44982025-04-30T06:25:40.736ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3cfefa3-b93a-4a2d-a1af-06554e04dcc8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
44992025-04-30T06:25:40.736ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
45002025-04-30T06:25:40.737ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f512692-d56b-4603-90c9-2e1fe5ac1fd3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45012025-04-30T06:25:40.737ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
45022025-04-30T06:25:40.738ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e7a32a8-a5d1-478c-aa61-e6ecb8c349ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45032025-04-30T06:25:40.739ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
45042025-04-30T06:25:40.740ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93626aa7-b0a3-4dbd-b4af-27f8fa9075be", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45052025-04-30T06:25:40.740ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
45062025-04-30T06:25:40.742ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4112337-bb7c-465f-b171-9e7164a0726f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45072025-04-30T06:25:40.743ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
45082025-04-30T06:25:40.745ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae47d075-57ed-4790-8936-f069a601097d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45092025-04-30T06:25:40.745ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
45102025-04-30T06:25:40.747ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ab99526-d4ce-454d-a058-d8ac4b084cc5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45112025-04-30T06:25:40.747ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
45122025-04-30T06:25:40.751ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a61e707c-1c9a-4a6a-9e28-23e9d4763417", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45132025-04-30T06:25:40.752ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
45142025-04-30T06:25:40.791ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45152025-04-30T06:25:40.791ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45162025-04-30T06:25:40.791ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45172025-04-30T06:25:40.791ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
45182025-04-30T06:25:40.792ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45192025-04-30T06:25:40.792ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
45202025-04-30T06:25:40.795ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45212025-04-30T06:25:40.816ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45222025-04-30T06:25:40.816ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45232025-04-30T06:25:40.816ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
45242025-04-30T06:25:40.816ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45252025-04-30T06:25:40.818ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6f296c8-36d7-4a50-b044-4457831bd8ea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45262025-04-30T06:25:40.818ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
45272025-04-30T06:25:40.819ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45282025-04-30T06:25:40.819ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
45292025-04-30T06:25:40.819ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45302025-04-30T06:25:40.824ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae916253-2039-4eab-9fc0-592a8ecc33fe", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45312025-04-30T06:25:40.824ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
45322025-04-30T06:25:40.828ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce3bbed8-1312-4f57-bcc4-18f5b152b360", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45332025-04-30T06:25:40.836ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
45342025-04-30T06:25:40.841ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a2c89d3-ebcd-40f7-8dcf-dc419c50af65", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45352025-04-30T06:25:40.841ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
45362025-04-30T06:25:40.842ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aea49208-1c37-4fef-a8da-b69a4271d248", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45372025-04-30T06:25:40.842ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
45382025-04-30T06:25:40.847ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39cdd70f-7191-418e-8fad-df9a9268f85b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45392025-04-30T06:25:40.847ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
45402025-04-30T06:25:40.850ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48482f6-8049-4743-9eed-4b291a7d82f0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45412025-04-30T06:25:40.850ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
45422025-04-30T06:25:40.853ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f53c7a-da70-4474-9e74-ced35494e4d6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45432025-04-30T06:25:40.853ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
45442025-04-30T06:25:40.857ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8e4df71-09c9-4a64-a9aa-ce0bb0f8ff38", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45452025-04-30T06:25:40.857ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
45462025-04-30T06:25:40.861ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
45472025-04-30T06:25:40.866ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "778ea24a-b6ff-4042-b213-34c074eda13b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45482025-04-30T06:25:40.866ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
45492025-04-30T06:25:40.869ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3f09f28-f6dd-460f-a8e4-68d26a7b4465", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45502025-04-30T06:25:40.870ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
45512025-04-30T06:25:40.873ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f9c5301-1c42-4f57-b4fa-18ce2abc4d69", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45522025-04-30T06:25:40.873ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
45532025-04-30T06:25:40.876ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edf4f5aa-e250-4b5f-ab55-86da728f81fd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45542025-04-30T06:25:40.877ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
45552025-04-30T06:25:40.880ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f10fc6c-18f9-49bb-aa85-17870f53244a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45562025-04-30T06:25:40.880ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
45572025-04-30T06:25:40.881ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77b7696c-9a6b-4d0f-aef2-581e56756a2e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45582025-04-30T06:25:40.882ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
45592025-04-30T06:25:40.883ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b583602-433e-4e3f-b4b5-258f9bfe1576", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45602025-04-30T06:25:40.883ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
45612025-04-30T06:25:40.884ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfe7cbe6-4f4a-4c60-acac-5284a7446c24", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45622025-04-30T06:25:40.884ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
45632025-04-30T06:25:40.885ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b690fdd-f84a-4af2-9984-3852c10d4814", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45642025-04-30T06:25:40.886ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
45652025-04-30T06:25:40.887ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2a92316-710e-42d3-b0d3-6494d35eab7c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45662025-04-30T06:25:40.887ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
45672025-04-30T06:25:40.889ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b84630d-ef4c-4a9a-9da0-9edf73f2f1cb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45682025-04-30T06:25:40.889ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
45692025-04-30T06:25:40.891ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a023b16a-d3b8-484e-ae22-5a7fee998787", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45702025-04-30T06:25:40.891ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
45712025-04-30T06:25:40.894ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "383102f2-19cd-46a0-b94c-be4bdcfcb84e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45722025-04-30T06:25:40.894ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
45732025-04-30T06:25:40.895ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6208893-b760-4819-9c21-bb47a65f9b73", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45742025-04-30T06:25:40.896ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
45752025-04-30T06:25:40.897ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f3816c-d39d-410a-abd6-fe0723f5eab5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45762025-04-30T06:25:40.897ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
45772025-04-30T06:25:40.899ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd04acbb-ac0f-4074-bb93-277f4073bc1d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45782025-04-30T06:25:40.899ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
45792025-04-30T06:25:40.900ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cfa35cc-e3c1-49f0-9684-ce0f1eaa385c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45802025-04-30T06:25:40.900ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
45812025-04-30T06:25:40.901ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a0e01f4-60c8-4ef0-962e-1ed1689c8f91", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45822025-04-30T06:25:40.901ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
45832025-04-30T06:25:40.902ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6284436f-bc2d-4a49-b285-81b053a404a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45842025-04-30T06:25:40.903ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
45852025-04-30T06:25:40.903ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0877db1d-e663-4c2f-b1de-622e5c877a84", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45862025-04-30T06:25:40.903ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
45872025-04-30T06:25:40.904ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
45882025-04-30T06:25:40.905ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e91f2ac-daa2-4665-b041-6c3cde887a64", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45892025-04-30T06:25:40.906ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
45902025-04-30T06:25:40.910ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c14dccd1-3dde-466a-a2cd-1dbfe0f846bf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45912025-04-30T06:25:40.910ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
45922025-04-30T06:25:40.912ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45932025-04-30T06:25:40.912ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45942025-04-30T06:25:40.912ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45952025-04-30T06:25:40.912ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
45962025-04-30T06:25:40.913ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45972025-04-30T06:25:40.914ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3d5cdf1-9e32-44a9-bdca-d7f0a65dbd05", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
45982025-04-30T06:25:40.914ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
45992025-04-30T06:25:40.917ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9fc2047-dbfc-44ce-9b3b-7ec18fa063a4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
46002025-04-30T06:25:40.917ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
46012025-04-30T06:25:40.917ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46022025-04-30T06:25:40.923ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46032025-04-30T06:25:40.923ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46042025-04-30T06:25:40.923ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
46052025-04-30T06:25:40.924ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46062025-04-30T06:25:40.928ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46072025-04-30T06:25:40.928ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
46082025-04-30T06:25:40.928ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46092025-04-30T06:25:40.930ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7572bbbf-75c4-42f3-98ef-2bf17c74da26", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
46102025-04-30T06:25:40.930ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46112025-04-30T06:25:40.930ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46122025-04-30T06:25:40.931ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46132025-04-30T06:25:40.931ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46142025-04-30T06:25:40.931ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46152025-04-30T06:25:40.937ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46162025-04-30T06:25:40.941ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46172025-04-30T06:25:40.941ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46182025-04-30T06:25:40.941ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46192025-04-30T06:25:40.941ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46202025-04-30T06:25:40.941ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46212025-04-30T06:25:40.943ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46222025-04-30T06:25:40.959ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46232025-04-30T06:25:40.962ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46242025-04-30T06:25:40.962ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46252025-04-30T06:25:40.962ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46262025-04-30T06:25:40.963ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46272025-04-30T06:25:41.014ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
46282025-04-30T06:25:41.014ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
46292025-04-30T06:25:41.015ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:59929 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
46302025-04-30T06:25:41.016ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:59929 req_id = ed2e2aba-4d94-4101-969a-a6067cb7357f sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
46312025-04-30T06:25:41.016ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 654 local_addr = [::1]:42668 method = GET remote_addr = [::1]:59929 req_id = ed2e2aba-4d94-4101-969a-a6067cb7357f response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
46322025-04-30T06:25:41.017ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed2e2aba-4d94-4101-969a-a6067cb7357f", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
46332025-04-30T06:25:41.079ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
46342025-04-30T06:25:41.079ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
46352025-04-30T06:25:41.080ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:51740 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
46362025-04-30T06:25:41.082ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:51740 req_id = 75a17545-0f9c-4deb-ba50-963a185c2f57 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
46372025-04-30T06:25:41.082ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 449 local_addr = [::1]:47918 method = GET remote_addr = [::1]:51740 req_id = 75a17545-0f9c-4deb-ba50-963a185c2f57 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
46382025-04-30T06:25:41.083ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "75a17545-0f9c-4deb-ba50-963a185c2f57", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
46392025-04-30T06:25:41.138ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
46402025-04-30T06:25:41.138ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
46412025-04-30T06:25:41.138ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:60870 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
46422025-04-30T06:25:41.139ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:60870 req_id = 59ef5306-5f50-4a68-bfde-f53c51f6895a sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
46432025-04-30T06:25:41.140ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 604 local_addr = [::1]:35744 method = GET remote_addr = [::1]:60870 req_id = 59ef5306-5f50-4a68-bfde-f53c51f6895a response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
46442025-04-30T06:25:41.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "59ef5306-5f50-4a68-bfde-f53c51f6895a", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
46452025-04-30T06:25:41.237ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
46462025-04-30T06:25:41.237ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
46472025-04-30T06:25:41.238ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:43853 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46482025-04-30T06:25:41.240ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:43853 req_id = 12349cb0-241c-41e6-9bee-8cfb2aa723bd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46492025-04-30T06:25:41.240ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 647 local_addr = [::1]:35343 method = GET remote_addr = [::1]:43853 req_id = 12349cb0-241c-41e6-9bee-8cfb2aa723bd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46502025-04-30T06:25:41.244ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "12349cb0-241c-41e6-9bee-8cfb2aa723bd", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:40 GMT"} })
46512025-04-30T06:25:41.247ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46522025-04-30T06:25:41.247ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46532025-04-30T06:25:41.247ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
46542025-04-30T06:25:41.250ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46552025-04-30T06:25:41.250ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46562025-04-30T06:25:41.250ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46572025-04-30T06:25:41.250ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46582025-04-30T06:25:41.251ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46592025-04-30T06:25:41.263ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46602025-04-30T06:25:41.266ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46612025-04-30T06:25:41.267ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46622025-04-30T06:25:41.267ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46632025-04-30T06:25:41.267ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46642025-04-30T06:25:41.268ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
46652025-04-30T06:25:41.312ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
46662025-04-30T06:25:41.317ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46672025-04-30T06:25:41.319ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 3370d1fe-066f-4221-ae49-4aa02d701303 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46682025-04-30T06:25:41.335ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 3370d1fe-066f-4221-ae49-4aa02d701303 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46692025-04-30T06:25:41.335ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 3370d1fe-066f-4221-ae49-4aa02d701303 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46702025-04-30T06:25:41.336ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 3370d1fe-066f-4221-ae49-4aa02d701303 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46712025-04-30T06:25:41.338ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46722025-04-30T06:25:41.338ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46732025-04-30T06:25:41.338ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46742025-04-30T06:25:41.338ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
46752025-04-30T06:25:41.340ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46762025-04-30T06:25:41.361ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 41228 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 3370d1fe-066f-4221-ae49-4aa02d701303 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
46772025-04-30T06:25:41.362ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "3370d1fe-066f-4221-ae49-4aa02d701303", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
46782025-04-30T06:25:41.362ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
46792025-04-30T06:25:41.362ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 49c79287-5194-4b10-9d47-02129f328585 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
46802025-04-30T06:25:41.364ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 49c79287-5194-4b10-9d47-02129f328585 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
46812025-04-30T06:25:41.364ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 49c79287-5194-4b10-9d47-02129f328585 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
46822025-04-30T06:25:41.365ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 49c79287-5194-4b10-9d47-02129f328585 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
46832025-04-30T06:25:41.366ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46842025-04-30T06:25:41.371ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46852025-04-30T06:25:41.371ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46862025-04-30T06:25:41.372ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
46872025-04-30T06:25:41.372ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46882025-04-30T06:25:41.375ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 12239 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 49c79287-5194-4b10-9d47-02129f328585 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
46892025-04-30T06:25:41.375ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "49c79287-5194-4b10-9d47-02129f328585", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
46902025-04-30T06:25:41.375ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
46912025-04-30T06:25:41.375ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
46922025-04-30T06:25:41.375ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
46932025-04-30T06:25:41.401ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
46942025-04-30T06:25:41.402ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
46952025-04-30T06:25:41.402ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 883.470568ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
46962025-04-30T06:25:41.402ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
46972025-04-30T06:25:41.402ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46982025-04-30T06:25:41.402ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46992025-04-30T06:25:41.402ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47002025-04-30T06:25:41.402ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47012025-04-30T06:25:41.403ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47022025-04-30T06:25:41.404ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47032025-04-30T06:25:41.404ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
47042025-04-30T06:25:41.404ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47052025-04-30T06:25:41.406ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47062025-04-30T06:25:41.414ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47072025-04-30T06:25:41.414ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47082025-04-30T06:25:41.414ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47092025-04-30T06:25:41.414ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47102025-04-30T06:25:41.418ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47112025-04-30T06:25:41.419ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47122025-04-30T06:25:41.419ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47132025-04-30T06:25:41.419ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47142025-04-30T06:25:41.419ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47152025-04-30T06:25:41.419ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47162025-04-30T06:25:41.419ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
47172025-04-30T06:25:41.419ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47182025-04-30T06:25:41.420ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 11 89 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65066 req_id = a6a3f42d-c5d5-4eaf-97ca-aa4e239e8b6f
47192025-04-30T06:25:41.420ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4489,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65066 req_id = a6a3f42d-c5d5-4eaf-97ca-aa4e239e8b6f
47202025-04-30T06:25:41.420ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
47212025-04-30T06:25:41.420ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
47222025-04-30T06:25:41.421ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47232025-04-30T06:25:41.421ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65066 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a6a3f42d-c5d5-4eaf-97ca-aa4e239e8b6f
47242025-04-30T06:25:41.432ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3206, tv_nsec: 231140836 } })
47252025-04-30T06:25:41.432ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
47262025-04-30T06:25:41.432ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
47272025-04-30T06:25:41.434ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e 6d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .m..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61076 req_id = 12c635ce-68e7-4fb1-8af1-0c742d93cc3d
47282025-04-30T06:25:41.434ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40557,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61076 req_id = 12c635ce-68e7-4fb1-8af1-0c742d93cc3d
47292025-04-30T06:25:41.434ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
47302025-04-30T06:25:41.434ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61076 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 12c635ce-68e7-4fb1-8af1-0c742d93cc3d
47312025-04-30T06:25:41.434ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7d b4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e }...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54645 req_id = 3ed378dd-df14-4049-8c9f-57a35e1075e5
47322025-04-30T06:25:41.434ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32180,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54645 req_id = 3ed378dd-df14-4049-8c9f-57a35e1075e5
47332025-04-30T06:25:41.434ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47342025-04-30T06:25:41.435ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54645 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3ed378dd-df14-4049-8c9f-57a35e1075e5
47352025-04-30T06:25:41.596ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47362025-04-30T06:25:41.605ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1a f5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61134 req_id = 1cd5796a-d79d-4822-a3ef-df058bf22665
47372025-04-30T06:25:41.605ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6901,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61134 req_id = 1cd5796a-d79d-4822-a3ef-df058bf22665
47382025-04-30T06:25:41.606ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
47392025-04-30T06:25:41.606ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6901, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61134 req_id = 1cd5796a-d79d-4822-a3ef-df058bf22665
47402025-04-30T06:25:41.607ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
47412025-04-30T06:25:41.607ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
47422025-04-30T06:25:41.607ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
47432025-04-30T06:25:41.621ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47442025-04-30T06:25:41.621ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47452025-04-30T06:25:41.621ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
47462025-04-30T06:25:41.627ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47472025-04-30T06:25:41.643ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
47482025-04-30T06:25:41.643ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47492025-04-30T06:25:41.643ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1610a474-a31b-419d-b45c-8f7abe1e3235", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47502025-04-30T06:25:41.644ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
47512025-04-30T06:25:41.652ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
47522025-04-30T06:25:41.652ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47532025-04-30T06:25:41.655ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
47542025-04-30T06:25:41.665ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1810e560-892c-4572-9d96-df2f0e7b8d3d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47552025-04-30T06:25:41.666ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
47562025-04-30T06:25:41.671ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d872312-c1e1-445f-8f0c-9ae78537f785", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47572025-04-30T06:25:41.672ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
47582025-04-30T06:25:41.674ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "454fcf87-4f03-4c23-9de4-2a8e0f2a0edb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47592025-04-30T06:25:41.675ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
47602025-04-30T06:25:41.677ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1654dda7-7164-41af-9b28-53987195161d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47612025-04-30T06:25:41.687ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
47622025-04-30T06:25:41.702ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74d31c33-c7af-4953-9101-4433555c8475", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47632025-04-30T06:25:41.702ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
47642025-04-30T06:25:41.704ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2756e3eb-6d67-4170-a80e-3e0fae8b54d6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47652025-04-30T06:25:41.704ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
47662025-04-30T06:25:41.706ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47672025-04-30T06:25:41.706ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47682025-04-30T06:25:41.706ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47692025-04-30T06:25:41.706ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
47702025-04-30T06:25:41.707ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47712025-04-30T06:25:41.712ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47722025-04-30T06:25:41.728ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47732025-04-30T06:25:41.729ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47742025-04-30T06:25:41.729ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
47752025-04-30T06:25:41.730ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47762025-04-30T06:25:41.731ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
47772025-04-30T06:25:41.733ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b1f67e7-2ec4-4a34-a34c-9231e261249c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47782025-04-30T06:25:41.733ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
47792025-04-30T06:25:41.734ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47802025-04-30T06:25:41.734ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
47812025-04-30T06:25:41.736ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47822025-04-30T06:25:41.739ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90b6b4e6-f250-47e0-a135-263bbfbd935a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47832025-04-30T06:25:41.739ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
47842025-04-30T06:25:41.746ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5363d4a-d146-49ca-bfb7-11a436b15085", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47852025-04-30T06:25:41.746ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
47862025-04-30T06:25:41.746ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "223dcb5f-085f-4a11-873f-048d0a0e10dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47872025-04-30T06:25:41.747ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
47882025-04-30T06:25:41.747ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "665c8b9b-fcb9-4e43-afa1-fd08091a7827", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47892025-04-30T06:25:41.747ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
47902025-04-30T06:25:41.749ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c6d109e-0956-4cd8-b886-91c0572dd5d5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47912025-04-30T06:25:41.749ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
47922025-04-30T06:25:41.757ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4157fef-8104-4e7e-846c-e45f598115d6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47932025-04-30T06:25:41.757ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
47942025-04-30T06:25:41.757ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65a6f68c-5b76-4ddb-80e0-e1371924eb35", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47952025-04-30T06:25:41.757ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
47962025-04-30T06:25:41.757ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6fd7070-260f-468f-88a7-63f74e74e527", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47972025-04-30T06:25:41.757ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
47982025-04-30T06:25:41.769ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "110d035a-3968-4164-aedd-c0be2c589900", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
47992025-04-30T06:25:41.769ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
48002025-04-30T06:25:41.774ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d853d686-7fc4-4e97-ba97-b51316fe4510", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48012025-04-30T06:25:41.774ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
48022025-04-30T06:25:41.776ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f56b6fc-8730-4e0c-90c0-eafc0615c9cb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48032025-04-30T06:25:41.776ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
48042025-04-30T06:25:41.777ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a4f8505-6d34-49a2-8f1c-fff99469f3d0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48052025-04-30T06:25:41.777ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
48062025-04-30T06:25:41.780ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "091b49d3-ea4f-4d61-ad7a-0388ab298a5a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48072025-04-30T06:25:41.780ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
48082025-04-30T06:25:41.784ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "264c5fec-3607-4eb4-9278-abc871572dcc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48092025-04-30T06:25:41.784ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
48102025-04-30T06:25:41.784ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
48112025-04-30T06:25:41.787ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35c51dd2-f8f5-4541-833b-f36a256d1b67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48122025-04-30T06:25:41.787ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
48132025-04-30T06:25:41.789ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "463afae6-1f53-4b27-a1c9-8af784765821", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48142025-04-30T06:25:41.789ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
48152025-04-30T06:25:41.794ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8350855-de2f-4f17-b6e0-eaff241b62e3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48162025-04-30T06:25:41.794ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
48172025-04-30T06:25:41.797ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "afeae2dc-8878-4904-a1d0-b10cc16446e6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48182025-04-30T06:25:41.798ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
48192025-04-30T06:25:41.801ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "625d471d-0078-4e36-9d0c-81bf9d55c0d3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48202025-04-30T06:25:41.801ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
48212025-04-30T06:25:41.806ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e9f61a-f464-4a63-b3bc-be3b19c89e88", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48222025-04-30T06:25:41.806ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
48232025-04-30T06:25:41.809ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1da702cb-b160-43d2-86e9-3acde31d0426", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48242025-04-30T06:25:41.809ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
48252025-04-30T06:25:41.813ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e38fad-9618-481a-b24a-330032c4d4c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48262025-04-30T06:25:41.813ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
48272025-04-30T06:25:41.814ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a37c31ea-936e-4f55-9937-6c63fda99218", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48282025-04-30T06:25:41.814ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
48292025-04-30T06:25:41.815ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9786a74-eaad-4787-92d3-0c154597017c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48302025-04-30T06:25:41.816ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
48312025-04-30T06:25:41.817ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde0c2f3-91a4-47c6-a1d5-f450cab78819", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48322025-04-30T06:25:41.818ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
48332025-04-30T06:25:41.819ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1db83b3d-4ad1-46d2-a51a-f61f11e28afc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48342025-04-30T06:25:41.819ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
48352025-04-30T06:25:41.821ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1aa1d80-5924-486a-b633-88ad52296b67", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48362025-04-30T06:25:41.821ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
48372025-04-30T06:25:41.824ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "057f9391-cbf6-435c-9d06-739267225d3b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48382025-04-30T06:25:41.824ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
48392025-04-30T06:25:41.826ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1192f3a9-947f-49b4-a286-0bcd02d3f5bc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48402025-04-30T06:25:41.827ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
48412025-04-30T06:25:41.829ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddf8f15f-e8b4-43f9-901b-1a0e610d86f7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48422025-04-30T06:25:41.830ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
48432025-04-30T06:25:41.833ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee43c830-3194-4652-9a9a-eaaa95ebfc9a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48442025-04-30T06:25:41.833ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
48452025-04-30T06:25:41.835ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48462025-04-30T06:25:41.835ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48472025-04-30T06:25:41.835ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48482025-04-30T06:25:41.835ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
48492025-04-30T06:25:41.836ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48502025-04-30T06:25:41.836ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "074658cf-fc4a-4dc8-8346-c387b04ef166", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48512025-04-30T06:25:41.836ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
48522025-04-30T06:25:41.838ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa7e9845-e481-45c7-9b0a-0ebbe134aa7e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48532025-04-30T06:25:41.838ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
48542025-04-30T06:25:41.839ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48552025-04-30T06:25:41.843ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48562025-04-30T06:25:41.843ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48572025-04-30T06:25:41.843ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
48582025-04-30T06:25:41.844ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48592025-04-30T06:25:41.848ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d8c689-3fe0-4529-b8e2-7a4d84743f4f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48602025-04-30T06:25:41.848ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
48612025-04-30T06:25:41.849ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3291787-ce49-4d1a-96c6-b1e11478101c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48622025-04-30T06:25:41.849ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
48632025-04-30T06:25:41.849ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48642025-04-30T06:25:41.849ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
48652025-04-30T06:25:41.850ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48662025-04-30T06:25:41.851ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9f61a8f-cd26-4b74-ae04-42125976387c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48672025-04-30T06:25:41.851ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
48682025-04-30T06:25:41.852ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a42951d-23e2-4f03-8a27-3bb59cddd9ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48692025-04-30T06:25:41.853ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
48702025-04-30T06:25:41.853ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
48712025-04-30T06:25:41.856ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecab3bd1-ba4e-47f6-b568-85aabb6f930b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48722025-04-30T06:25:41.856ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
48732025-04-30T06:25:41.858ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4099537-9277-442d-bcfd-1c72fc3a9ef5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48742025-04-30T06:25:41.858ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
48752025-04-30T06:25:41.859ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ce040c0-d8c3-4eb9-84e4-89545cbb2db0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48762025-04-30T06:25:41.859ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
48772025-04-30T06:25:41.860ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b48d64a9-24ee-4e90-8417-d6b225a683ac", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48782025-04-30T06:25:41.860ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
48792025-04-30T06:25:41.862ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0db1d79e-7e81-422c-80c1-3116c5a116df", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
48802025-04-30T06:25:41.862ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48812025-04-30T06:25:41.862ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48822025-04-30T06:25:41.862ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48832025-04-30T06:25:41.862ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48842025-04-30T06:25:41.863ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48852025-04-30T06:25:41.869ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48862025-04-30T06:25:41.873ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48872025-04-30T06:25:41.873ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48882025-04-30T06:25:41.873ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48892025-04-30T06:25:41.873ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48902025-04-30T06:25:41.873ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48912025-04-30T06:25:41.873ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48922025-04-30T06:25:41.876ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48932025-04-30T06:25:41.881ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48942025-04-30T06:25:41.881ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48952025-04-30T06:25:41.882ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48962025-04-30T06:25:41.883ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48972025-04-30T06:25:41.951ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
48982025-04-30T06:25:41.951ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
48992025-04-30T06:25:41.952ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:44605 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
49002025-04-30T06:25:41.953ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:44605 req_id = 1bb860f9-edee-42ec-abfb-e798b14254da sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
49012025-04-30T06:25:41.953ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 446 local_addr = [::1]:42668 method = GET remote_addr = [::1]:44605 req_id = 1bb860f9-edee-42ec-abfb-e798b14254da response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
49022025-04-30T06:25:41.954ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bb860f9-edee-42ec-abfb-e798b14254da", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
49032025-04-30T06:25:42.024ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
49042025-04-30T06:25:42.024ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
49052025-04-30T06:25:42.025ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:58118 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
49062025-04-30T06:25:42.026ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:58118 req_id = 92df3a06-4a14-4298-b1ae-92b1415528b8 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
49072025-04-30T06:25:42.027ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 861 local_addr = [::1]:47918 method = GET remote_addr = [::1]:58118 req_id = 92df3a06-4a14-4298-b1ae-92b1415528b8 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
49082025-04-30T06:25:42.031ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "92df3a06-4a14-4298-b1ae-92b1415528b8", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
49092025-04-30T06:25:42.100ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
49102025-04-30T06:25:42.100ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
49112025-04-30T06:25:42.101ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
49122025-04-30T06:25:42.102ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:59009 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
49132025-04-30T06:25:42.107ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:59009 req_id = 5313f99f-2ffe-43d3-854a-7fd28913b399 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
49142025-04-30T06:25:42.108ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 643 local_addr = [::1]:35744 method = GET remote_addr = [::1]:59009 req_id = 5313f99f-2ffe-43d3-854a-7fd28913b399 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
49152025-04-30T06:25:42.108ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5313f99f-2ffe-43d3-854a-7fd28913b399", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
49162025-04-30T06:25:42.169ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
49172025-04-30T06:25:42.169ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
49182025-04-30T06:25:42.170ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49192025-04-30T06:25:42.170ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49202025-04-30T06:25:42.170ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49212025-04-30T06:25:42.170ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
49222025-04-30T06:25:42.171ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49232025-04-30T06:25:42.171ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:50635 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
49242025-04-30T06:25:42.172ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:50635 req_id = 5b333daf-cdb6-4c54-8939-d6aae39d74e3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49252025-04-30T06:25:42.173ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 584 local_addr = [::1]:35343 method = GET remote_addr = [::1]:50635 req_id = 5b333daf-cdb6-4c54-8939-d6aae39d74e3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49262025-04-30T06:25:42.174ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b333daf-cdb6-4c54-8939-d6aae39d74e3", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:41 GMT"} })
49272025-04-30T06:25:42.178ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49282025-04-30T06:25:42.178ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49292025-04-30T06:25:42.178ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
49302025-04-30T06:25:42.183ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49312025-04-30T06:25:42.183ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49322025-04-30T06:25:42.184ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49332025-04-30T06:25:42.184ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49342025-04-30T06:25:42.186ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49352025-04-30T06:25:42.187ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49362025-04-30T06:25:42.193ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49372025-04-30T06:25:42.193ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49382025-04-30T06:25:42.193ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
49392025-04-30T06:25:42.194ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49402025-04-30T06:25:42.204ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49412025-04-30T06:25:42.208ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49422025-04-30T06:25:42.208ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49432025-04-30T06:25:42.208ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49442025-04-30T06:25:42.209ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49452025-04-30T06:25:42.211ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49462025-04-30T06:25:42.211ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
49472025-04-30T06:25:42.212ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49482025-04-30T06:25:42.310ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
49492025-04-30T06:25:42.328ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49502025-04-30T06:25:42.329ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49512025-04-30T06:25:42.329ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 927.083624ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49522025-04-30T06:25:42.329ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
49532025-04-30T06:25:42.329ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49542025-04-30T06:25:42.329ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49552025-04-30T06:25:42.329ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49562025-04-30T06:25:42.329ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49572025-04-30T06:25:42.331ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49582025-04-30T06:25:42.335ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49592025-04-30T06:25:42.341ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49602025-04-30T06:25:42.341ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49612025-04-30T06:25:42.341ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49622025-04-30T06:25:42.342ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49632025-04-30T06:25:42.345ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
49642025-04-30T06:25:42.345ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49652025-04-30T06:25:42.346ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 94 e8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54086 req_id = 7e2b67e9-b7a9-4682-9012-cb485001f958
49662025-04-30T06:25:42.346ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38120,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54086 req_id = 7e2b67e9-b7a9-4682-9012-cb485001f958
49672025-04-30T06:25:42.346ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
49682025-04-30T06:25:42.346ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
49692025-04-30T06:25:42.346ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49702025-04-30T06:25:42.346ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54086 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7e2b67e9-b7a9-4682-9012-cb485001f958
49712025-04-30T06:25:42.347ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3207, tv_nsec: 146250672 } })
49722025-04-30T06:25:42.347ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
49732025-04-30T06:25:42.347ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
49742025-04-30T06:25:42.347ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 62 f1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e b...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55265 req_id = 8f7b72d8-9187-4b1e-8b83-18aa94ee960e
49752025-04-30T06:25:42.347ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25329,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55265 req_id = 8f7b72d8-9187-4b1e-8b83-18aa94ee960e
49762025-04-30T06:25:42.348ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
49772025-04-30T06:25:42.348ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55265 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8f7b72d8-9187-4b1e-8b83-18aa94ee960e
49782025-04-30T06:25:42.348ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9f d8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62837 req_id = 74bfec9e-523b-4c24-9957-aca6394db94d
49792025-04-30T06:25:42.348ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40920,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62837 req_id = 74bfec9e-523b-4c24-9957-aca6394db94d
49802025-04-30T06:25:42.348ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49812025-04-30T06:25:42.348ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62837 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 74bfec9e-523b-4c24-9957-aca6394db94d
49822025-04-30T06:25:42.438ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49832025-04-30T06:25:42.438ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49842025-04-30T06:25:42.438ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49852025-04-30T06:25:42.438ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49862025-04-30T06:25:42.438ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
49872025-04-30T06:25:42.439ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49882025-04-30T06:25:42.439ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49892025-04-30T06:25:42.440ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49902025-04-30T06:25:42.442ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f7 62 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .b..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58989 req_id = 0008f182-6b82-4d8b-8ad7-ea619acb81fd
49912025-04-30T06:25:42.442ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63330,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58989 req_id = 0008f182-6b82-4d8b-8ad7-ea619acb81fd
49922025-04-30T06:25:42.442ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
49932025-04-30T06:25:42.443ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 63330, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58989 req_id = 0008f182-6b82-4d8b-8ad7-ea619acb81fd
49942025-04-30T06:25:42.449ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
49952025-04-30T06:25:42.449ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
49962025-04-30T06:25:42.449ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
49972025-04-30T06:25:42.450ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49982025-04-30T06:25:42.451ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
49992025-04-30T06:25:42.451ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50002025-04-30T06:25:42.451ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
50012025-04-30T06:25:42.451ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50022025-04-30T06:25:42.457ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50032025-04-30T06:25:42.457ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50042025-04-30T06:25:42.457ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
50052025-04-30T06:25:42.458ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50062025-04-30T06:25:42.467ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50072025-04-30T06:25:42.467ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
50082025-04-30T06:25:42.468ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50092025-04-30T06:25:42.481ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "26345062-34df-4c90-92f1-98c12a0b695e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50102025-04-30T06:25:42.481ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
50112025-04-30T06:25:42.494ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dd9fd8f-a256-492b-a91f-64f5a44bff64", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50122025-04-30T06:25:42.494ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
50132025-04-30T06:25:42.511ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf8a397d-54b4-4d9f-8c36-1ab124255859", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50142025-04-30T06:25:42.511ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
50152025-04-30T06:25:42.520ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c67daa-3fd9-4b25-8ec4-4b03b9f47e00", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50162025-04-30T06:25:42.520ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
50172025-04-30T06:25:42.525ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c16be455-342a-4898-bfab-4b47cf5a22b5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50182025-04-30T06:25:42.525ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
50192025-04-30T06:25:42.528ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82c6c1c4-b2ac-4ba8-9ca6-ae282ada7147", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50202025-04-30T06:25:42.528ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
50212025-04-30T06:25:42.536ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
50222025-04-30T06:25:42.546ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ace5f888-9b7a-4177-81c0-6822e175e0a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50232025-04-30T06:25:42.546ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
50242025-04-30T06:25:42.549ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd2c2d6-ea42-4df2-9d18-0a3754c501c0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50252025-04-30T06:25:42.549ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
50262025-04-30T06:25:42.550ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58bf8f1f-f37c-4696-af07-b08fa1aa26e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50272025-04-30T06:25:42.551ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
50282025-04-30T06:25:42.552ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb8d9f7-1671-415c-9ddf-f40f5b0bab78", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50292025-04-30T06:25:42.553ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
50302025-04-30T06:25:42.554ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b54d67a-1121-4b25-9e66-c0946570fe75", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50312025-04-30T06:25:42.554ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
50322025-04-30T06:25:42.555ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eda2d77-e5e2-4f92-b9fb-926fb452912c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50332025-04-30T06:25:42.555ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
50342025-04-30T06:25:42.556ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77d30c87-5085-4fde-9eca-606343f71d4d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50352025-04-30T06:25:42.557ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
50362025-04-30T06:25:42.559ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d60ce0-ba32-4f54-958c-571b395e3941", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50372025-04-30T06:25:42.559ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
50382025-04-30T06:25:42.561ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "039a726c-b5bd-4c4e-a907-32125b9da6dd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50392025-04-30T06:25:42.562ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
50402025-04-30T06:25:42.564ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69635a82-78fc-49ec-89f8-64a47538e1c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50412025-04-30T06:25:42.564ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
50422025-04-30T06:25:42.566ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4c0f689-fae2-4855-8847-0d484d3f9719", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50432025-04-30T06:25:42.566ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
50442025-04-30T06:25:42.569ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46d215f0-807a-45ae-ace9-41af27a50308", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50452025-04-30T06:25:42.569ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
50462025-04-30T06:25:42.571ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88193490-2b35-4da4-a87f-c4a1f1d71823", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50472025-04-30T06:25:42.571ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
50482025-04-30T06:25:42.573ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "464be2cc-20df-49ea-8db4-0ae138e28f55", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50492025-04-30T06:25:42.573ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
50502025-04-30T06:25:42.574ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd85cf45-285c-4333-b577-bf6203f03b08", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50512025-04-30T06:25:42.574ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
50522025-04-30T06:25:42.576ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c853fd3-5c2d-4b78-a8e8-d6b3fc42ff7d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50532025-04-30T06:25:42.576ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
50542025-04-30T06:25:42.577ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4675295-1b18-4a83-bc8f-61f88624530d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50552025-04-30T06:25:42.577ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
50562025-04-30T06:25:42.580ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca4c7329-6f35-484d-9101-34d2cf9d82c1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50572025-04-30T06:25:42.580ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
50582025-04-30T06:25:42.582ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5129301-bc9b-40f0-9e76-f395a1fad41a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50592025-04-30T06:25:42.582ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
50602025-04-30T06:25:42.603ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50612025-04-30T06:25:42.603ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50622025-04-30T06:25:42.603ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50632025-04-30T06:25:42.603ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
50642025-04-30T06:25:42.604ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50652025-04-30T06:25:42.606ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a46a2b4-ce19-4515-98d5-5ddb9c042f73", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50662025-04-30T06:25:42.606ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
50672025-04-30T06:25:42.610ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50682025-04-30T06:25:42.610ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50692025-04-30T06:25:42.610ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
50702025-04-30T06:25:42.611ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
50712025-04-30T06:25:42.614ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50722025-04-30T06:25:42.623ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50732025-04-30T06:25:42.623ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50742025-04-30T06:25:42.623ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
50752025-04-30T06:25:42.624ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50762025-04-30T06:25:42.628ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50772025-04-30T06:25:42.628ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
50782025-04-30T06:25:42.629ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50792025-04-30T06:25:42.629ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
50802025-04-30T06:25:42.629ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
50812025-04-30T06:25:42.630ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f77729aa-9ebb-4623-a54b-346e397cf4e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50822025-04-30T06:25:42.631ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
50832025-04-30T06:25:42.635ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdf9430e-6cb8-47ce-8622-58aa2ba37e1f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50842025-04-30T06:25:42.635ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
50852025-04-30T06:25:42.636ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
50862025-04-30T06:25:42.636ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
50872025-04-30T06:25:42.638ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b649e2d-5404-4847-9abb-24946a7cc258", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50882025-04-30T06:25:42.638ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
50892025-04-30T06:25:42.641ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eac8ee4-b974-4477-b82f-aea7af3bab7e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50902025-04-30T06:25:42.641ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
50912025-04-30T06:25:42.644ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6859377f-d183-42b5-a049-7b2ec134eef6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50922025-04-30T06:25:42.644ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
50932025-04-30T06:25:42.647ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0539afd-3272-4cc0-ac0b-a358e01db841", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50942025-04-30T06:25:42.647ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
50952025-04-30T06:25:42.649ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d216b5fd-2bf8-4a30-9cd0-80719773a8e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50962025-04-30T06:25:42.649ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
50972025-04-30T06:25:42.651ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac73eb6-071d-4fc7-aeea-a4fbd528fa83", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
50982025-04-30T06:25:42.651ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
50992025-04-30T06:25:42.657ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19320f46-acf5-4684-9078-9d4049ef80b9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51002025-04-30T06:25:42.657ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
51012025-04-30T06:25:42.661ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b17e70a6-fc55-4b81-bc69-98c4816da695", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51022025-04-30T06:25:42.661ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
51032025-04-30T06:25:42.669ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
51042025-04-30T06:25:42.670ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3e4a716-1acd-4423-b425-293fa8124f0d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51052025-04-30T06:25:42.676ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
51062025-04-30T06:25:42.679ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0455e1be-6c18-4639-aac1-0206ad05ee52", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51072025-04-30T06:25:42.679ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
51082025-04-30T06:25:42.680ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e55704b-9f0a-4bbe-adce-d24afe58fbf5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51092025-04-30T06:25:42.681ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
51102025-04-30T06:25:42.682ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e286df6-4f06-4756-90d8-1b5db1b2c395", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51112025-04-30T06:25:42.683ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
51122025-04-30T06:25:42.686ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d7f40eb-aee0-4946-9f60-ad790523c7cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51132025-04-30T06:25:42.686ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
51142025-04-30T06:25:42.687ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "226f150c-1898-4485-8969-b3b9109b0044", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51152025-04-30T06:25:42.687ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
51162025-04-30T06:25:42.691ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8107ce0a-eeeb-4724-a8a9-25b8c599ae0f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51172025-04-30T06:25:42.691ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
51182025-04-30T06:25:42.692ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af444684-af16-4c5b-96c4-9149b84ac4db", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51192025-04-30T06:25:42.692ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
51202025-04-30T06:25:42.693ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb4ee7e3-2f80-4ddc-aa11-785cf488f73c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51212025-04-30T06:25:42.693ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
51222025-04-30T06:25:42.693ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
51232025-04-30T06:25:42.694ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c5745a-d591-4be0-8995-cade86f1d940", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51242025-04-30T06:25:42.695ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
51252025-04-30T06:25:42.698ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af51edec-16a8-49b7-8c51-92b9b6fac553", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51262025-04-30T06:25:42.698ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
51272025-04-30T06:25:42.698ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "82301f9b-d5f4-48ef-8325-d5d66687ab56", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51282025-04-30T06:25:42.699ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
51292025-04-30T06:25:42.706ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "abe365d9-6dd5-442d-b1d7-f3377eee8565", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51302025-04-30T06:25:42.706ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
51312025-04-30T06:25:42.708ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abe093cc-fbd6-4e36-891e-004e0ee64c51", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51322025-04-30T06:25:42.708ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51332025-04-30T06:25:42.708ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51342025-04-30T06:25:42.708ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51352025-04-30T06:25:42.708ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51362025-04-30T06:25:42.708ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51372025-04-30T06:25:42.714ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
51382025-04-30T06:25:42.714ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51392025-04-30T06:25:42.724ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51402025-04-30T06:25:42.724ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
51412025-04-30T06:25:42.724ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51422025-04-30T06:25:42.724ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51432025-04-30T06:25:42.724ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51442025-04-30T06:25:42.724ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51452025-04-30T06:25:42.725ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51462025-04-30T06:25:42.725ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51472025-04-30T06:25:42.725ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51482025-04-30T06:25:42.725ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
51492025-04-30T06:25:42.726ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51502025-04-30T06:25:42.728ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51512025-04-30T06:25:42.731ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
51522025-04-30T06:25:42.731ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51532025-04-30T06:25:42.731ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51542025-04-30T06:25:42.732ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51552025-04-30T06:25:42.734ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51562025-04-30T06:25:42.740ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51572025-04-30T06:25:42.740ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51582025-04-30T06:25:42.741ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
51592025-04-30T06:25:42.741ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51602025-04-30T06:25:42.810ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
51612025-04-30T06:25:42.810ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
51622025-04-30T06:25:42.811ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:35980 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
51632025-04-30T06:25:42.812ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:35980 req_id = eb15096c-e494-463c-893a-5adb023f0640 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
51642025-04-30T06:25:42.812ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 564 local_addr = [::1]:42668 method = GET remote_addr = [::1]:35980 req_id = eb15096c-e494-463c-893a-5adb023f0640 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
51652025-04-30T06:25:42.812ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51662025-04-30T06:25:42.812ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
51672025-04-30T06:25:42.814ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51682025-04-30T06:25:42.815ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb15096c-e494-463c-893a-5adb023f0640", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51692025-04-30T06:25:42.870ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
51702025-04-30T06:25:42.870ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
51712025-04-30T06:25:42.872ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:57120 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
51722025-04-30T06:25:42.876ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:57120 req_id = 4a592562-8ad0-4598-9740-81beac60a231 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
51732025-04-30T06:25:42.876ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 361 local_addr = [::1]:47918 method = GET remote_addr = [::1]:57120 req_id = 4a592562-8ad0-4598-9740-81beac60a231 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
51742025-04-30T06:25:42.877ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a592562-8ad0-4598-9740-81beac60a231", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51752025-04-30T06:25:42.933ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
51762025-04-30T06:25:42.933ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
51772025-04-30T06:25:42.934ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:32822 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
51782025-04-30T06:25:42.934ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:32822 req_id = 8666f95e-a0ff-48f5-bc2b-3ff844db4fa8 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
51792025-04-30T06:25:42.934ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 267 local_addr = [::1]:35744 method = GET remote_addr = [::1]:32822 req_id = 8666f95e-a0ff-48f5-bc2b-3ff844db4fa8 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
51802025-04-30T06:25:42.935ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8666f95e-a0ff-48f5-bc2b-3ff844db4fa8", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51812025-04-30T06:25:42.988ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
51822025-04-30T06:25:42.988ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
51832025-04-30T06:25:42.990ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:47279 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51842025-04-30T06:25:42.992ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:47279 req_id = f7c4a29f-b2eb-4d9b-89ec-4dfd18754605 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51852025-04-30T06:25:42.992ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 492 local_addr = [::1]:35343 method = GET remote_addr = [::1]:47279 req_id = f7c4a29f-b2eb-4d9b-89ec-4dfd18754605 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51862025-04-30T06:25:42.992ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c4a29f-b2eb-4d9b-89ec-4dfd18754605", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
51872025-04-30T06:25:42.994ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51882025-04-30T06:25:42.994ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51892025-04-30T06:25:42.994ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
51902025-04-30T06:25:43.001ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51912025-04-30T06:25:43.001ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51922025-04-30T06:25:43.001ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51932025-04-30T06:25:43.001ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51942025-04-30T06:25:43.001ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51952025-04-30T06:25:43.017ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51962025-04-30T06:25:43.021ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51972025-04-30T06:25:43.022ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51982025-04-30T06:25:43.022ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51992025-04-30T06:25:43.022ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52002025-04-30T06:25:43.062ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
52012025-04-30T06:25:43.078ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
52022025-04-30T06:25:43.078ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
52032025-04-30T06:25:43.078ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 749.030484ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
52042025-04-30T06:25:43.078ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
52052025-04-30T06:25:43.078ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
52062025-04-30T06:25:43.078ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52072025-04-30T06:25:43.078ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52082025-04-30T06:25:43.078ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52092025-04-30T06:25:43.079ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52102025-04-30T06:25:43.086ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52112025-04-30T06:25:43.091ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
52122025-04-30T06:25:43.091ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
52132025-04-30T06:25:43.091ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
52142025-04-30T06:25:43.091ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
52152025-04-30T06:25:43.097ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
52162025-04-30T06:25:43.097ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
52172025-04-30T06:25:43.098ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b1 91 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53577 req_id = 6bea8abf-2ddd-4a8a-bca7-4303032cc913
52182025-04-30T06:25:43.098ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45457,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53577 req_id = 6bea8abf-2ddd-4a8a-bca7-4303032cc913
52192025-04-30T06:25:43.098ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
52202025-04-30T06:25:43.099ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
52212025-04-30T06:25:43.099ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
52222025-04-30T06:25:43.099ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53577 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6bea8abf-2ddd-4a8a-bca7-4303032cc913
52232025-04-30T06:25:43.100ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3207, tv_nsec: 899304843 } })
52242025-04-30T06:25:43.100ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
52252025-04-30T06:25:43.100ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
52262025-04-30T06:25:43.101ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e5 2e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52403 req_id = faeafeab-0108-4289-810c-19f4900c9b3c
52272025-04-30T06:25:43.101ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58670,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52403 req_id = faeafeab-0108-4289-810c-19f4900c9b3c
52282025-04-30T06:25:43.101ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
52292025-04-30T06:25:43.101ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52403 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = faeafeab-0108-4289-810c-19f4900c9b3c
52302025-04-30T06:25:43.102ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: df e4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61432 req_id = 7ccec035-0f24-4d9c-8476-9eda45164808
52312025-04-30T06:25:43.102ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57316,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61432 req_id = 7ccec035-0f24-4d9c-8476-9eda45164808
52322025-04-30T06:25:43.102ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
52332025-04-30T06:25:43.102ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61432 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7ccec035-0f24-4d9c-8476-9eda45164808
52342025-04-30T06:25:43.234ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
52352025-04-30T06:25:43.234ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52362025-04-30T06:25:43.234ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52372025-04-30T06:25:43.234ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52382025-04-30T06:25:43.235ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
52392025-04-30T06:25:43.236ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52402025-04-30T06:25:43.237ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5d dc 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ]............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:50156 req_id = e10cd61a-c03b-4d42-ad71-cfec9432617e
52412025-04-30T06:25:43.238ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24028,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50156 req_id = e10cd61a-c03b-4d42-ad71-cfec9432617e
52422025-04-30T06:25:43.238ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
52432025-04-30T06:25:43.238ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24028, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:50156 req_id = e10cd61a-c03b-4d42-ad71-cfec9432617e
52442025-04-30T06:25:43.239ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
52452025-04-30T06:25:43.239ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
52462025-04-30T06:25:43.239ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
52472025-04-30T06:25:43.241ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52482025-04-30T06:25:43.246ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52492025-04-30T06:25:43.246ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52502025-04-30T06:25:43.246ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
52512025-04-30T06:25:43.246ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52522025-04-30T06:25:43.268ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52532025-04-30T06:25:43.269ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
52542025-04-30T06:25:43.269ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52552025-04-30T06:25:43.270ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3c46882-e1d1-4d0b-ae37-d765dd76b5dd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52562025-04-30T06:25:43.271ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
52572025-04-30T06:25:43.287ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59f2a0f7-a879-49e0-900d-26c0233d321b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52582025-04-30T06:25:43.288ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
52592025-04-30T06:25:43.298ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9410c2e-ed2e-4cdc-ad66-692c62a3f6fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52602025-04-30T06:25:43.298ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
52612025-04-30T06:25:43.298ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
52622025-04-30T06:25:43.299ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52632025-04-30T06:25:43.300ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 31f66103-b9ba-4297-81e2-fde1a977a2ff uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52642025-04-30T06:25:43.300ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 31f66103-b9ba-4297-81e2-fde1a977a2ff resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52652025-04-30T06:25:43.300ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 31f66103-b9ba-4297-81e2-fde1a977a2ff roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52662025-04-30T06:25:43.301ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 31f66103-b9ba-4297-81e2-fde1a977a2ff resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52672025-04-30T06:25:43.304ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 4494 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 31f66103-b9ba-4297-81e2-fde1a977a2ff response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
52682025-04-30T06:25:43.305ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "31f66103-b9ba-4297-81e2-fde1a977a2ff", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52692025-04-30T06:25:43.305ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
52702025-04-30T06:25:43.305ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50032b19-4021-4edc-a43e-2807e5c2ae96", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52712025-04-30T06:25:43.306ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
52722025-04-30T06:25:43.306ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 7c946b52-a472-4199-aa63-54b3c798aeb8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
52732025-04-30T06:25:43.307ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 7c946b52-a472-4199-aa63-54b3c798aeb8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
52742025-04-30T06:25:43.307ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 7c946b52-a472-4199-aa63-54b3c798aeb8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
52752025-04-30T06:25:43.308ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 7c946b52-a472-4199-aa63-54b3c798aeb8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
52762025-04-30T06:25:43.311ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 4235 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 7c946b52-a472-4199-aa63-54b3c798aeb8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
52772025-04-30T06:25:43.312ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c946b52-a472-4199-aa63-54b3c798aeb8", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52782025-04-30T06:25:43.312ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52792025-04-30T06:25:43.312ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
52802025-04-30T06:25:43.312ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
52812025-04-30T06:25:43.316ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b6dacae-6494-4d45-bb8a-0c2ecf46a9f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52822025-04-30T06:25:43.316ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
52832025-04-30T06:25:43.317ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
52842025-04-30T06:25:43.320ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee573a41-7e8c-4a19-b6a0-6136426299f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52852025-04-30T06:25:43.320ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
52862025-04-30T06:25:43.324ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c8ce726-c6e9-449b-8223-0c10faa058d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52872025-04-30T06:25:43.324ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
52882025-04-30T06:25:43.333ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9bcec61-1b5e-432e-9b30-c69df4e9acef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52892025-04-30T06:25:43.333ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
52902025-04-30T06:25:43.334ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cd43237-2a87-4483-a58b-d4dda9da0c60", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52912025-04-30T06:25:43.334ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
52922025-04-30T06:25:43.336ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "799b716e-fa3f-41bf-98bb-6864b618d284", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52932025-04-30T06:25:43.336ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
52942025-04-30T06:25:43.337ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10f6c7f6-e2b6-4407-bd32-cffc64fe8ca2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52952025-04-30T06:25:43.337ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
52962025-04-30T06:25:43.339ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e2e7dd0-eeae-4a42-9889-923a580ce404", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52972025-04-30T06:25:43.339ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
52982025-04-30T06:25:43.341ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b637aab-a1a3-4a17-adcf-6e46ab1d75f5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
52992025-04-30T06:25:43.342ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
53002025-04-30T06:25:43.343ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c880ff8-eea6-4058-bc38-0ce8c8018191", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53012025-04-30T06:25:43.343ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
53022025-04-30T06:25:43.346ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "583aef9a-0d40-438a-9892-974f91f1daa2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53032025-04-30T06:25:43.347ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
53042025-04-30T06:25:43.348ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fa43cb2-a71b-4689-b6d0-a127b9cd73b0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53052025-04-30T06:25:43.348ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
53062025-04-30T06:25:43.351ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b948b04-0754-482d-8cec-ebc1e8a42d89", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53072025-04-30T06:25:43.351ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
53082025-04-30T06:25:43.354ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ef61016-0b1d-4974-860a-55103f747ab9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53092025-04-30T06:25:43.354ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
53102025-04-30T06:25:43.357ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd1edec4-2dd0-4a15-a840-574a24417a8e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53112025-04-30T06:25:43.357ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
53122025-04-30T06:25:43.358ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eba7a0b5-9c77-4760-a20a-a24c51b658c9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53132025-04-30T06:25:43.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
53142025-04-30T06:25:43.360ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9733e0d3-3efc-4231-8533-4c99ffdb516a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53152025-04-30T06:25:43.360ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
53162025-04-30T06:25:43.361ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22a14278-30b7-4824-8a25-0b55863f4461", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53172025-04-30T06:25:43.361ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
53182025-04-30T06:25:43.362ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6fb160f-88d0-477b-a7ae-1560e56e8b37", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53192025-04-30T06:25:43.362ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
53202025-04-30T06:25:43.364ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7f2e9fe-0461-4d73-ba5c-fd09655ba19b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53212025-04-30T06:25:43.364ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
53222025-04-30T06:25:43.366ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec55363b-9b85-44f4-99f2-f841153ba1a9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53232025-04-30T06:25:43.366ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
53242025-04-30T06:25:43.368ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53252025-04-30T06:25:43.368ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53262025-04-30T06:25:43.369ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53272025-04-30T06:25:43.369ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
53282025-04-30T06:25:43.369ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53292025-04-30T06:25:43.373ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53302025-04-30T06:25:43.386ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53312025-04-30T06:25:43.387ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53322025-04-30T06:25:43.387ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
53332025-04-30T06:25:43.387ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53342025-04-30T06:25:43.388ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a760a3da-86e2-4b1d-abc0-77e032ed8bc8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53352025-04-30T06:25:43.388ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
53362025-04-30T06:25:43.391ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53372025-04-30T06:25:43.391ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
53382025-04-30T06:25:43.392ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53392025-04-30T06:25:43.392ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f170da5-acfd-4464-9120-54aed2970329", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53402025-04-30T06:25:43.393ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
53412025-04-30T06:25:43.396ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "634d0ac8-9da3-4952-afb4-1c520109ec3f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53422025-04-30T06:25:43.396ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
53432025-04-30T06:25:43.400ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed76b1dc-d185-42ef-825b-3e29e41c2639", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53442025-04-30T06:25:43.401ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
53452025-04-30T06:25:43.403ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dfafec7-6a56-47c6-8f83-9d9ed96bc944", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53462025-04-30T06:25:43.403ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
53472025-04-30T06:25:43.406ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3c567f3-50f7-4152-b764-e3ba3b0b8624", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53482025-04-30T06:25:43.406ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
53492025-04-30T06:25:43.409ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0a9d29b-fe12-42b4-9e33-92b2bd6cf405", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53502025-04-30T06:25:43.409ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
53512025-04-30T06:25:43.410ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53522025-04-30T06:25:43.411ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53532025-04-30T06:25:43.413ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53542025-04-30T06:25:43.414ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53552025-04-30T06:25:43.414ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53562025-04-30T06:25:43.414ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
53572025-04-30T06:25:43.415ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b7b6456-f823-4cfd-aa90-fe8091b6f09a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53582025-04-30T06:25:43.415ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
53592025-04-30T06:25:43.416ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57e4d194-8e6e-4ebb-a674-3cf38cbdf747", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53602025-04-30T06:25:43.417ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
53612025-04-30T06:25:43.418ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54636d91-b81a-4e98-a0cd-a12c3aec04b4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53622025-04-30T06:25:43.418ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
53632025-04-30T06:25:43.420ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7f7dddb-29c1-4da4-b6cf-84de6824985e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53642025-04-30T06:25:43.420ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
53652025-04-30T06:25:43.424ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
53662025-04-30T06:25:43.426ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e9c733c-4b73-42b7-85b1-32f46ac3bc19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53672025-04-30T06:25:43.426ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
53682025-04-30T06:25:43.427ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "152b0292-f26b-4d48-ad48-76c443da675e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:42 GMT"} })
53692025-04-30T06:25:43.428ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
53702025-04-30T06:25:43.464ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32d74fd0-ad87-48a9-8b11-24f6e7fb433b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53712025-04-30T06:25:43.464ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
53722025-04-30T06:25:43.470ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f70a3036-d733-47b4-8491-ec0bb13bd4cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53732025-04-30T06:25:43.470ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
53742025-04-30T06:25:43.476ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dfe5064-4ada-4679-924e-29ec09f7c187", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53752025-04-30T06:25:43.478ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
53762025-04-30T06:25:43.479ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53772025-04-30T06:25:43.479ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53782025-04-30T06:25:43.479ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53792025-04-30T06:25:43.480ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
53802025-04-30T06:25:43.481ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53812025-04-30T06:25:43.483ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c49b5e72-8a62-47fb-ac36-2294768a80b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53822025-04-30T06:25:43.484ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
53832025-04-30T06:25:43.485ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "193a49b4-99b5-4b10-9c7e-72b417f84857", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53842025-04-30T06:25:43.485ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
53852025-04-30T06:25:43.485ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53862025-04-30T06:25:43.491ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53872025-04-30T06:25:43.491ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53882025-04-30T06:25:43.491ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
53892025-04-30T06:25:43.492ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53902025-04-30T06:25:43.494ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "022f72e0-3d81-4c81-923e-265d455d7c34", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53912025-04-30T06:25:43.494ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
53922025-04-30T06:25:43.495ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53932025-04-30T06:25:43.495ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
53942025-04-30T06:25:43.499ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53952025-04-30T06:25:43.501ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e43ff8f-f587-483e-8304-0413aee91351", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53962025-04-30T06:25:43.501ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
53972025-04-30T06:25:43.501ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
53982025-04-30T06:25:43.505ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d065715-1209-4ad1-8e44-9acc23b2bd52", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
53992025-04-30T06:25:43.505ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
54002025-04-30T06:25:43.507ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f65566-e8ad-4d41-9b90-fbbd224b76e5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54012025-04-30T06:25:43.507ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
54022025-04-30T06:25:43.508ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a144872-8588-4ba3-a480-fc29fa891a3d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54032025-04-30T06:25:43.509ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
54042025-04-30T06:25:43.510ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad60b7b2-63ff-465f-b57a-e5c280a2d552", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54052025-04-30T06:25:43.510ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
54062025-04-30T06:25:43.512ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ce763e-8e89-4da2-9bd4-28380a284289", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54072025-04-30T06:25:43.512ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
54082025-04-30T06:25:43.513ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54092025-04-30T06:25:43.513ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54102025-04-30T06:25:43.513ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54112025-04-30T06:25:43.514ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54122025-04-30T06:25:43.518ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54132025-04-30T06:25:43.525ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
54142025-04-30T06:25:43.525ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
54152025-04-30T06:25:43.526ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54162025-04-30T06:25:43.526ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54172025-04-30T06:25:43.526ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54182025-04-30T06:25:43.526ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54192025-04-30T06:25:43.531ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54202025-04-30T06:25:43.534ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
54212025-04-30T06:25:43.534ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54222025-04-30T06:25:43.534ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54232025-04-30T06:25:43.535ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54242025-04-30T06:25:43.587ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
54252025-04-30T06:25:43.587ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
54262025-04-30T06:25:43.588ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:63045 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
54272025-04-30T06:25:43.589ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:63045 req_id = ff1384b8-73c8-4d73-8d61-6f8d214b6fe1 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
54282025-04-30T06:25:43.589ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 403 local_addr = [::1]:42668 method = GET remote_addr = [::1]:63045 req_id = ff1384b8-73c8-4d73-8d61-6f8d214b6fe1 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
54292025-04-30T06:25:43.589ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff1384b8-73c8-4d73-8d61-6f8d214b6fe1", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54302025-04-30T06:25:43.668ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
54312025-04-30T06:25:43.668ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
54322025-04-30T06:25:43.669ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:34530 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
54332025-04-30T06:25:43.669ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54342025-04-30T06:25:43.669ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54352025-04-30T06:25:43.669ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
54362025-04-30T06:25:43.670ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
54372025-04-30T06:25:43.671ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:34530 req_id = d50934f5-a52d-4fd7-8756-7a6bdbd478eb sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
54382025-04-30T06:25:43.671ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 461 local_addr = [::1]:47918 method = GET remote_addr = [::1]:34530 req_id = d50934f5-a52d-4fd7-8756-7a6bdbd478eb response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
54392025-04-30T06:25:43.672ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d50934f5-a52d-4fd7-8756-7a6bdbd478eb", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54402025-04-30T06:25:43.742ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
54412025-04-30T06:25:43.742ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
54422025-04-30T06:25:43.746ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:41401 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
54432025-04-30T06:25:43.747ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:41401 req_id = e236a744-8d80-43f5-a7eb-465d3afe2b71 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
54442025-04-30T06:25:43.747ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 300 local_addr = [::1]:35744 method = GET remote_addr = [::1]:41401 req_id = e236a744-8d80-43f5-a7eb-465d3afe2b71 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
54452025-04-30T06:25:43.748ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
54462025-04-30T06:25:43.749ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e236a744-8d80-43f5-a7eb-465d3afe2b71", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54472025-04-30T06:25:43.820ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
54482025-04-30T06:25:43.820ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
54492025-04-30T06:25:43.822ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54502025-04-30T06:25:43.822ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54512025-04-30T06:25:43.822ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54522025-04-30T06:25:43.822ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
54532025-04-30T06:25:43.824ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54542025-04-30T06:25:43.824ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:62708 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
54552025-04-30T06:25:43.826ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
54562025-04-30T06:25:43.826ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
54572025-04-30T06:25:43.826ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
54582025-04-30T06:25:43.827ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:62708 req_id = d11f1c87-c97c-4a0b-aece-f43250db788b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
54592025-04-30T06:25:43.827ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 840 local_addr = [::1]:35343 method = GET remote_addr = [::1]:62708 req_id = d11f1c87-c97c-4a0b-aece-f43250db788b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
54602025-04-30T06:25:43.828ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d11f1c87-c97c-4a0b-aece-f43250db788b", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
54612025-04-30T06:25:43.833ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
54622025-04-30T06:25:43.833ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
54632025-04-30T06:25:43.833ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
54642025-04-30T06:25:43.838ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
54652025-04-30T06:25:43.838ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54662025-04-30T06:25:43.838ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54672025-04-30T06:25:43.839ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54682025-04-30T06:25:43.839ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54692025-04-30T06:25:43.841ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
54702025-04-30T06:25:43.841ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
54712025-04-30T06:25:43.843ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54722025-04-30T06:25:43.850ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54732025-04-30T06:25:43.850ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54742025-04-30T06:25:43.850ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
54752025-04-30T06:25:43.850ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54762025-04-30T06:25:43.851ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54772025-04-30T06:25:43.857ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54782025-04-30T06:25:43.858ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54792025-04-30T06:25:43.858ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54802025-04-30T06:25:43.858ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54812025-04-30T06:25:43.860ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54822025-04-30T06:25:43.860ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
54832025-04-30T06:25:43.861ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54842025-04-30T06:25:43.900ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
54852025-04-30T06:25:43.900ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
54862025-04-30T06:25:43.900ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 821.719813ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
54872025-04-30T06:25:43.900ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
54882025-04-30T06:25:43.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
54892025-04-30T06:25:43.900ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54902025-04-30T06:25:43.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54912025-04-30T06:25:43.900ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54922025-04-30T06:25:43.901ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54932025-04-30T06:25:43.904ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54942025-04-30T06:25:43.911ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
54952025-04-30T06:25:43.911ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
54962025-04-30T06:25:43.911ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
54972025-04-30T06:25:43.912ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
54982025-04-30T06:25:43.912ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
54992025-04-30T06:25:43.920ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
55002025-04-30T06:25:43.920ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55012025-04-30T06:25:43.926ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3b 4d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ;M..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54958 req_id = abbfaca0-8700-4cc1-8e28-2efac2ce0477
55022025-04-30T06:25:43.927ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15181,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54958 req_id = abbfaca0-8700-4cc1-8e28-2efac2ce0477
55032025-04-30T06:25:43.927ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
55042025-04-30T06:25:43.927ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
55052025-04-30T06:25:43.927ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55062025-04-30T06:25:43.927ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54958 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = abbfaca0-8700-4cc1-8e28-2efac2ce0477
55072025-04-30T06:25:43.928ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3208, tv_nsec: 727019581 } })
55082025-04-30T06:25:43.928ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
55092025-04-30T06:25:43.928ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
55102025-04-30T06:25:43.928ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: eb c1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63214 req_id = 62ee9d37-aba2-4908-a700-630c41aa68e8
55112025-04-30T06:25:43.928ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60353,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63214 req_id = 62ee9d37-aba2-4908-a700-630c41aa68e8
55122025-04-30T06:25:43.928ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
55132025-04-30T06:25:43.928ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63214 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 62ee9d37-aba2-4908-a700-630c41aa68e8
55142025-04-30T06:25:43.928ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0a 19 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64316 req_id = bb31c106-bd1a-4f4e-8a71-44a4b9702b4b
55152025-04-30T06:25:43.929ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2585,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64316 req_id = bb31c106-bd1a-4f4e-8a71-44a4b9702b4b
55162025-04-30T06:25:43.929ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55172025-04-30T06:25:43.929ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64316 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bb31c106-bd1a-4f4e-8a71-44a4b9702b4b
55182025-04-30T06:25:44.043ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
55192025-04-30T06:25:44.043ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55202025-04-30T06:25:44.044ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55212025-04-30T06:25:44.044ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55222025-04-30T06:25:44.044ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
55232025-04-30T06:25:44.045ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55242025-04-30T06:25:44.045ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ff d8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64895 req_id = 48deb3a7-1dca-4464-a4c5-7ae367e004dc
55252025-04-30T06:25:44.045ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65496,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64895 req_id = 48deb3a7-1dca-4464-a4c5-7ae367e004dc
55262025-04-30T06:25:44.045ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
55272025-04-30T06:25:44.045ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65496, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64895 req_id = 48deb3a7-1dca-4464-a4c5-7ae367e004dc
55282025-04-30T06:25:44.046ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
55292025-04-30T06:25:44.046ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
55302025-04-30T06:25:44.046ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
55312025-04-30T06:25:44.051ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55322025-04-30T06:25:44.061ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55332025-04-30T06:25:44.061ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55342025-04-30T06:25:44.061ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
55352025-04-30T06:25:44.062ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55362025-04-30T06:25:44.067ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b985d0ed-b4b1-4b12-bd7b-bb6173827555", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55372025-04-30T06:25:44.068ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
55382025-04-30T06:25:44.069ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55392025-04-30T06:25:44.069ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
55402025-04-30T06:25:44.070ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55412025-04-30T06:25:44.070ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0ef144f-58a7-460c-a944-f93115a43261", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55422025-04-30T06:25:44.070ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
55432025-04-30T06:25:44.077ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "226748a9-1fd3-42cd-bfa7-9638e18af4cc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55442025-04-30T06:25:44.079ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
55452025-04-30T06:25:44.083ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a30f7c6c-2a9d-4a98-af86-7b7a44d39253", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55462025-04-30T06:25:44.083ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
55472025-04-30T06:25:44.088ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d46340ab-9841-45f1-b151-593abc81dd7a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55482025-04-30T06:25:44.088ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
55492025-04-30T06:25:44.092ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d2fbf3-fe8b-4dcc-a0d1-0e22cb664d14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55502025-04-30T06:25:44.093ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
55512025-04-30T06:25:44.100ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4e91a49-9b00-4aa4-ad02-dade9ec5dd4c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55522025-04-30T06:25:44.100ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
55532025-04-30T06:25:44.114ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a73f756-bd16-4ca1-85e3-2b5ffc0a19cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55542025-04-30T06:25:44.115ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
55552025-04-30T06:25:44.117ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f2ebba7-595c-43cf-941b-89d86e5fb953", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55562025-04-30T06:25:44.117ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
55572025-04-30T06:25:44.118ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
55582025-04-30T06:25:44.120ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b751ee0-9c24-47cb-9074-7af857d1ab52", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55592025-04-30T06:25:44.120ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
55602025-04-30T06:25:44.121ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfe8c056-3df3-4b4d-94ff-06aa4b406ca6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55612025-04-30T06:25:44.122ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
55622025-04-30T06:25:44.123ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "822d21eb-af57-4162-a762-ec4997593409", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55632025-04-30T06:25:44.123ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
55642025-04-30T06:25:44.124ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e86c40b0-4b27-465f-a91b-2b940e18a56b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55652025-04-30T06:25:44.125ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
55662025-04-30T06:25:44.126ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ad20a11-17b4-4a84-9da6-b74ec677da68", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55672025-04-30T06:25:44.126ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
55682025-04-30T06:25:44.128ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9e77256-c8f5-4d7e-b9d7-7eefc0fdafe9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55692025-04-30T06:25:44.129ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
55702025-04-30T06:25:44.130ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8d60a80-139d-415a-baf8-f2f04f16077f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55712025-04-30T06:25:44.138ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
55722025-04-30T06:25:44.140ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7a6364-2d8d-4638-aaf5-e1c789ac650f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55732025-04-30T06:25:44.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
55742025-04-30T06:25:44.169ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55752025-04-30T06:25:44.170ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55762025-04-30T06:25:44.170ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55772025-04-30T06:25:44.170ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
55782025-04-30T06:25:44.171ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55792025-04-30T06:25:44.179ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55802025-04-30T06:25:44.197ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55812025-04-30T06:25:44.197ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55822025-04-30T06:25:44.197ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
55832025-04-30T06:25:44.198ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55842025-04-30T06:25:44.198ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75a3d152-ee1b-4ceb-ba35-625a1840aadf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55852025-04-30T06:25:44.199ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
55862025-04-30T06:25:44.211ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55872025-04-30T06:25:44.211ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
55882025-04-30T06:25:44.212ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55892025-04-30T06:25:44.221ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b72ac17-562c-4011-8b2f-fb1dc19274bf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55902025-04-30T06:25:44.221ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
55912025-04-30T06:25:44.225ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcb42e00-6ae8-4e62-b2a3-371daeee01e1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55922025-04-30T06:25:44.225ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
55932025-04-30T06:25:44.227ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4be4bb2-017a-4b71-bc0f-ec51dcb5bf60", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55942025-04-30T06:25:44.228ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
55952025-04-30T06:25:44.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88defc9a-e2e2-41d8-8af7-ed634d64a827", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55962025-04-30T06:25:44.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
55972025-04-30T06:25:44.234ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9fe518-93d8-41d5-ad0f-a434e4a2f5e6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
55982025-04-30T06:25:44.234ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
55992025-04-30T06:25:44.236ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "593b41ab-9595-42a3-9de5-aa35d270aae3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56002025-04-30T06:25:44.236ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
56012025-04-30T06:25:44.239ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1347d027-747a-4f0d-86ce-c20b3f631451", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56022025-04-30T06:25:44.239ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
56032025-04-30T06:25:44.242ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c506027-92d9-4cea-8571-6a06c633597b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56042025-04-30T06:25:44.242ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
56052025-04-30T06:25:44.245ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdcd875f-012c-49d3-a2d2-ca35dd7a4cb0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56062025-04-30T06:25:44.245ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
56072025-04-30T06:25:44.246ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25242229-c8ed-4d90-a9a1-e440c61b9f1b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56082025-04-30T06:25:44.247ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
56092025-04-30T06:25:44.259ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
56102025-04-30T06:25:44.264ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cffa0152-0e84-4178-9768-f2c625519eb6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56112025-04-30T06:25:44.264ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
56122025-04-30T06:25:44.268ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61878f81-1795-42cd-b6df-852aece90673", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56132025-04-30T06:25:44.268ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
56142025-04-30T06:25:44.270ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93cafe31-947a-468b-b1f8-d22d6d2641a3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56152025-04-30T06:25:44.270ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
56162025-04-30T06:25:44.271ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2072a3bd-3767-4414-b708-80d5d9c6b496", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56172025-04-30T06:25:44.272ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
56182025-04-30T06:25:44.273ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c01b056-2cf9-45dc-a4b0-68ff2bcda7da", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56192025-04-30T06:25:44.273ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
56202025-04-30T06:25:44.274ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afd6fed6-7e65-4390-b631-da31c85520c5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56212025-04-30T06:25:44.274ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
56222025-04-30T06:25:44.276ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae7bcd94-825e-4496-9bb6-1846a322d048", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56232025-04-30T06:25:44.277ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
56242025-04-30T06:25:44.279ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52dba0bc-9e2c-4f0c-b540-eda983cfcb06", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56252025-04-30T06:25:44.279ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
56262025-04-30T06:25:44.281ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91024ff5-c363-4d0a-bb18-03115fee5263", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56272025-04-30T06:25:44.281ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
56282025-04-30T06:25:44.283ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f74d66d5-66c1-474c-98b2-c1e79fad75e7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56292025-04-30T06:25:44.283ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
56302025-04-30T06:25:44.285ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f732ef31-e9ec-4671-9350-130f17f1f916", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56312025-04-30T06:25:44.285ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
56322025-04-30T06:25:44.286ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46558b0b-5e53-4e25-8e25-ebec5a9d72db", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56332025-04-30T06:25:44.286ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
56342025-04-30T06:25:44.290ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d04dda0d-e959-47e5-9af9-ad60c757a481", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56352025-04-30T06:25:44.290ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
56362025-04-30T06:25:44.290ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "160dcb16-978a-4897-b16e-4d7f7c448f8c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56372025-04-30T06:25:44.291ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
56382025-04-30T06:25:44.291ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80f76a2f-ebe3-4aaf-8f95-ca4ff31eb027", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56392025-04-30T06:25:44.291ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
56402025-04-30T06:25:44.292ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0341204-1a8a-44d8-91da-57ee6aca15ea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56412025-04-30T06:25:44.293ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
56422025-04-30T06:25:44.294ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a19a7a-6dac-4e2e-a1b4-5fae3a523aab", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56432025-04-30T06:25:44.294ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
56442025-04-30T06:25:44.294ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
56452025-04-30T06:25:44.298ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b39a83e3-0876-49f7-b9db-ed0efae2a3ba", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56462025-04-30T06:25:44.299ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
56472025-04-30T06:25:44.301ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "735f2662-12fd-4883-bf67-595af7da2a43", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56482025-04-30T06:25:44.302ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
56492025-04-30T06:25:44.304ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d67322e-c452-475d-bb6a-6c1d115943ef", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56502025-04-30T06:25:44.304ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
56512025-04-30T06:25:44.306ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a12a33-b621-4cb1-9262-2ce80e3abcad", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56522025-04-30T06:25:44.307ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
56532025-04-30T06:25:44.309ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8194887-bcf8-46a2-a12b-5a6aa3bc5a17", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56542025-04-30T06:25:44.309ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
56552025-04-30T06:25:44.309ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56562025-04-30T06:25:44.309ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56572025-04-30T06:25:44.310ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56582025-04-30T06:25:44.311ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56592025-04-30T06:25:44.311ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56602025-04-30T06:25:44.311ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56612025-04-30T06:25:44.311ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56622025-04-30T06:25:44.312ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
56632025-04-30T06:25:44.313ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56642025-04-30T06:25:44.316ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56652025-04-30T06:25:44.324ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56662025-04-30T06:25:44.324ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56672025-04-30T06:25:44.324ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
56682025-04-30T06:25:44.325ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56692025-04-30T06:25:44.325ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56702025-04-30T06:25:44.329ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
56712025-04-30T06:25:44.329ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
56722025-04-30T06:25:44.329ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56732025-04-30T06:25:44.329ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56742025-04-30T06:25:44.330ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56752025-04-30T06:25:44.332ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56762025-04-30T06:25:44.332ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56772025-04-30T06:25:44.332ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
56782025-04-30T06:25:44.333ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56792025-04-30T06:25:44.333ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56802025-04-30T06:25:44.338ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
56812025-04-30T06:25:44.338ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
56822025-04-30T06:25:44.338ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
56832025-04-30T06:25:44.339ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
56842025-04-30T06:25:44.448ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
56852025-04-30T06:25:44.448ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
56862025-04-30T06:25:44.454ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:34200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
56872025-04-30T06:25:44.455ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56882025-04-30T06:25:44.456ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56892025-04-30T06:25:44.461ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:34200 req_id = 485ed378-971c-4aa3-a439-e308a4b79b4e sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
56902025-04-30T06:25:44.463ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2052 local_addr = [::1]:42668 method = GET remote_addr = [::1]:34200 req_id = 485ed378-971c-4aa3-a439-e308a4b79b4e response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
56912025-04-30T06:25:44.469ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "485ed378-971c-4aa3-a439-e308a4b79b4e", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:43 GMT"} })
56922025-04-30T06:25:44.542ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
56932025-04-30T06:25:44.542ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
56942025-04-30T06:25:44.542ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56952025-04-30T06:25:44.544ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56962025-04-30T06:25:44.544ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56972025-04-30T06:25:44.544ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56982025-04-30T06:25:44.546ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:54786 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
56992025-04-30T06:25:44.548ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:54786 req_id = 2c31ab8f-b918-4a7e-becf-caedde55bb22 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
57002025-04-30T06:25:44.556ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 8209 local_addr = [::1]:47918 method = GET remote_addr = [::1]:54786 req_id = 2c31ab8f-b918-4a7e-becf-caedde55bb22 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
57012025-04-30T06:25:44.558ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c31ab8f-b918-4a7e-becf-caedde55bb22", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57022025-04-30T06:25:44.655ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
57032025-04-30T06:25:44.655ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
57042025-04-30T06:25:44.656ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:38244 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
57052025-04-30T06:25:44.657ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57062025-04-30T06:25:44.657ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57072025-04-30T06:25:44.664ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
57082025-04-30T06:25:44.664ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
57092025-04-30T06:25:44.666ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:38244 req_id = 361ade23-7948-44d3-8a57-205f5463f9cd sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
57102025-04-30T06:25:44.667ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 795 local_addr = [::1]:35744 method = GET remote_addr = [::1]:38244 req_id = 361ade23-7948-44d3-8a57-205f5463f9cd response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
57112025-04-30T06:25:44.672ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "361ade23-7948-44d3-8a57-205f5463f9cd", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57122025-04-30T06:25:44.736ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
57132025-04-30T06:25:44.736ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
57142025-04-30T06:25:44.740ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:45374 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57152025-04-30T06:25:44.746ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
57162025-04-30T06:25:44.746ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
57172025-04-30T06:25:44.749ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:45374 req_id = 527b2827-872f-42cd-a52f-561bbebd09d4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57182025-04-30T06:25:44.762ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 12887 local_addr = [::1]:35343 method = GET remote_addr = [::1]:45374 req_id = 527b2827-872f-42cd-a52f-561bbebd09d4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57192025-04-30T06:25:44.765ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "527b2827-872f-42cd-a52f-561bbebd09d4", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57202025-04-30T06:25:44.770ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57212025-04-30T06:25:44.772ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57222025-04-30T06:25:44.773ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
57232025-04-30T06:25:44.780ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57242025-04-30T06:25:44.780ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57252025-04-30T06:25:44.780ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57262025-04-30T06:25:44.780ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57272025-04-30T06:25:44.781ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57282025-04-30T06:25:44.787ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
57292025-04-30T06:25:44.787ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57302025-04-30T06:25:44.790ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
57312025-04-30T06:25:44.793ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
57322025-04-30T06:25:44.796ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57332025-04-30T06:25:44.811ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57342025-04-30T06:25:44.812ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57352025-04-30T06:25:44.812ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57362025-04-30T06:25:44.813ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57372025-04-30T06:25:44.849ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57382025-04-30T06:25:44.849ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57392025-04-30T06:25:44.849ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57402025-04-30T06:25:44.849ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
57412025-04-30T06:25:44.850ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57422025-04-30T06:25:44.853ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57432025-04-30T06:25:44.859ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57442025-04-30T06:25:44.859ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57452025-04-30T06:25:44.859ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
57462025-04-30T06:25:44.861ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57472025-04-30T06:25:44.888ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57482025-04-30T06:25:44.897ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
57492025-04-30T06:25:44.897ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57502025-04-30T06:25:44.897ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57512025-04-30T06:25:44.898ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57522025-04-30T06:25:44.898ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 997.839781ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57532025-04-30T06:25:44.898ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
57542025-04-30T06:25:44.898ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57552025-04-30T06:25:44.898ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57562025-04-30T06:25:44.898ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57572025-04-30T06:25:44.898ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57582025-04-30T06:25:44.900ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57592025-04-30T06:25:44.908ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57602025-04-30T06:25:44.912ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57612025-04-30T06:25:44.912ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57622025-04-30T06:25:44.912ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57632025-04-30T06:25:44.912ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57642025-04-30T06:25:44.915ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
57652025-04-30T06:25:44.916ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
57662025-04-30T06:25:44.916ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 90 a8 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57085 req_id = 86f9898e-898e-43c9-949a-8b5a84aec31b
57672025-04-30T06:25:44.916ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37032,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57085 req_id = 86f9898e-898e-43c9-949a-8b5a84aec31b
57682025-04-30T06:25:44.917ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
57692025-04-30T06:25:44.917ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
57702025-04-30T06:25:44.917ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57712025-04-30T06:25:44.917ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57085 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 86f9898e-898e-43c9-949a-8b5a84aec31b
57722025-04-30T06:25:44.919ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3209, tv_nsec: 718026658 } })
57732025-04-30T06:25:44.919ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
57742025-04-30T06:25:44.919ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
57752025-04-30T06:25:44.920ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1f 32 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .2..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51710 req_id = 4c3b9f29-c65b-4f95-b3d4-eff046b0e068
57762025-04-30T06:25:44.920ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7986,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51710 req_id = 4c3b9f29-c65b-4f95-b3d4-eff046b0e068
57772025-04-30T06:25:44.920ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
57782025-04-30T06:25:44.920ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51710 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4c3b9f29-c65b-4f95-b3d4-eff046b0e068
57792025-04-30T06:25:44.920ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a5 b1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60467 req_id = c0047cab-df46-4fe6-9a3f-cccc48d53adf
57802025-04-30T06:25:44.920ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42417,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60467 req_id = c0047cab-df46-4fe6-9a3f-cccc48d53adf
57812025-04-30T06:25:44.920ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
57822025-04-30T06:25:44.920ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60467 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c0047cab-df46-4fe6-9a3f-cccc48d53adf
57832025-04-30T06:25:45.039ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
57842025-04-30T06:25:45.039ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1c 22 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ."..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57424 req_id = cfa75ab7-e20b-40ad-8350-c6ea10de1612
57852025-04-30T06:25:45.040ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7202,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57424 req_id = cfa75ab7-e20b-40ad-8350-c6ea10de1612
57862025-04-30T06:25:45.040ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
57872025-04-30T06:25:45.040ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7202, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57424 req_id = cfa75ab7-e20b-40ad-8350-c6ea10de1612
57882025-04-30T06:25:45.040ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
57892025-04-30T06:25:45.040ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
57902025-04-30T06:25:45.040ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
57912025-04-30T06:25:45.043ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f15cc83-b7a4-47b5-8d07-12c04664db4d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57922025-04-30T06:25:45.043ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
57932025-04-30T06:25:45.044ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6179c10b-11c7-461c-9c85-48845a7480db", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57942025-04-30T06:25:45.044ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
57952025-04-30T06:25:45.054ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e112950-2c08-4756-8ae6-eb87405dad14", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57962025-04-30T06:25:45.054ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
57972025-04-30T06:25:45.060ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
57982025-04-30T06:25:45.063ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30172f61-dbdb-4564-95f3-939f792eadce", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
57992025-04-30T06:25:45.063ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
58002025-04-30T06:25:45.066ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "759a4291-e8b6-40eb-8679-9e9b9ebd1edc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58012025-04-30T06:25:45.066ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
58022025-04-30T06:25:45.068ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45535c88-77a2-4e39-9c09-d77a922b1ee4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58032025-04-30T06:25:45.068ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
58042025-04-30T06:25:45.071ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e65acd2-d1a4-41b4-a8d7-9e7c3b784af5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58052025-04-30T06:25:45.071ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
58062025-04-30T06:25:45.074ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03bd8ace-07f0-4533-8491-2aa7fedfb14b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58072025-04-30T06:25:45.074ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
58082025-04-30T06:25:45.075ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e62d24b-754a-458a-9689-5d24c4fdbcb4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58092025-04-30T06:25:45.075ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
58102025-04-30T06:25:45.077ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe164fd4-8f4e-47f8-bd7b-699ce5ea4328", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58112025-04-30T06:25:45.077ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
58122025-04-30T06:25:45.079ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca59a3ad-1e83-4176-affb-c2c807aefabd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58132025-04-30T06:25:45.079ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
58142025-04-30T06:25:45.080ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "892f3d54-5cbe-4bc9-8814-e84dad0791e1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58152025-04-30T06:25:45.080ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
58162025-04-30T06:25:45.086ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bda40e2-9220-41fa-94ea-ac493f94cd0d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58172025-04-30T06:25:45.087ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
58182025-04-30T06:25:45.088ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e11a15d0-10de-4099-aa3b-2b7aa6d5d7f2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58192025-04-30T06:25:45.089ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
58202025-04-30T06:25:45.091ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d548ae9a-1947-4ebc-80e9-bdbac35f1b4b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58212025-04-30T06:25:45.091ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
58222025-04-30T06:25:45.102ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12daf6d8-c983-4b66-aa7b-c11237628c0f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58232025-04-30T06:25:45.102ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
58242025-04-30T06:25:45.104ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b40d165-0bcc-49c5-840d-86ffc181d276", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58252025-04-30T06:25:45.104ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
58262025-04-30T06:25:45.106ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7141608c-64eb-4bc9-abec-38ba2ff1e614", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58272025-04-30T06:25:45.107ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
58282025-04-30T06:25:45.109ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8448bb0-253f-4f23-8cd2-d61692aa4534", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58292025-04-30T06:25:45.109ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
58302025-04-30T06:25:45.109ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b47e0270-b939-4fdd-b946-a1a70f76469a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58312025-04-30T06:25:45.110ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
58322025-04-30T06:25:45.111ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58332025-04-30T06:25:45.111ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58342025-04-30T06:25:45.111ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58352025-04-30T06:25:45.111ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
58362025-04-30T06:25:45.112ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58372025-04-30T06:25:45.112ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77bcaa7d-9c17-4bcd-9578-114ee4673738", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58382025-04-30T06:25:45.112ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
58392025-04-30T06:25:45.114ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4edc8959-5b69-4afc-9cc5-288c79bff8a1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58402025-04-30T06:25:45.114ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
58412025-04-30T06:25:45.114ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58422025-04-30T06:25:45.119ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58432025-04-30T06:25:45.119ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58442025-04-30T06:25:45.119ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
58452025-04-30T06:25:45.119ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58462025-04-30T06:25:45.121ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8879b4e5-ea8d-4946-84e4-530b7cf2ab9c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58472025-04-30T06:25:45.121ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
58482025-04-30T06:25:45.122ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e276bbe-8e9d-4c01-852f-50f52bd53f25", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58492025-04-30T06:25:45.122ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
58502025-04-30T06:25:45.122ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58512025-04-30T06:25:45.122ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
58522025-04-30T06:25:45.123ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58532025-04-30T06:25:45.125ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5527879-917c-4543-963a-759e438d3608", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58542025-04-30T06:25:45.125ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
58552025-04-30T06:25:45.127ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d494a39-8ef2-43af-8649-3a5705a12711", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58562025-04-30T06:25:45.127ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
58572025-04-30T06:25:45.130ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "488c3f30-ba9a-400f-bdb1-8fc109d71290", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58582025-04-30T06:25:45.130ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
58592025-04-30T06:25:45.134ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c1e8153-8e1f-4112-930a-8963a98755c4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58602025-04-30T06:25:45.134ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
58612025-04-30T06:25:45.138ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f69af03-5136-4388-8f66-9474da67baea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58622025-04-30T06:25:45.138ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
58632025-04-30T06:25:45.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a15d07d7-fb22-4c38-955c-ba5b21e812fd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58642025-04-30T06:25:45.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
58652025-04-30T06:25:45.142ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69da5aca-0bfc-4541-b5a2-08142dc0344f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58662025-04-30T06:25:45.142ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
58672025-04-30T06:25:45.153ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9d9a592-ea05-4f4a-aeb0-4e7e72326e5b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58682025-04-30T06:25:45.153ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
58692025-04-30T06:25:45.161ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
58702025-04-30T06:25:45.165ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8ce66bb-162b-4cec-b36c-ee30779d003c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58712025-04-30T06:25:45.165ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
58722025-04-30T06:25:45.171ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ed3b2e7-d7ee-4536-ad09-61499e98941e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58732025-04-30T06:25:45.171ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
58742025-04-30T06:25:45.178ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abf637b5-d2a8-42cb-91f6-b7ae31960f6d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58752025-04-30T06:25:45.178ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
58762025-04-30T06:25:45.197ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "218f3b37-a616-4b59-8b2f-863c6ea7369f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58772025-04-30T06:25:45.197ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
58782025-04-30T06:25:45.203ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b63dbff4-3de2-402e-bcc4-548d0e2cd0ae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58792025-04-30T06:25:45.203ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
58802025-04-30T06:25:45.205ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cbb4edd-8724-453f-ab8c-df581cf11eac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58812025-04-30T06:25:45.205ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
58822025-04-30T06:25:45.208ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17088012-bbbf-40b2-80da-d1ae933895e9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58832025-04-30T06:25:45.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
58842025-04-30T06:25:45.211ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76dd51ab-3cd3-47d2-a585-b2b9596bf796", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58852025-04-30T06:25:45.211ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
58862025-04-30T06:25:45.212ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58872025-04-30T06:25:45.212ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58882025-04-30T06:25:45.212ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58892025-04-30T06:25:45.212ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
58902025-04-30T06:25:45.213ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58912025-04-30T06:25:45.216ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "481df58f-6b7a-495c-967d-2e1278f2904f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58922025-04-30T06:25:45.216ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
58932025-04-30T06:25:45.217ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58942025-04-30T06:25:45.226ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58952025-04-30T06:25:45.226ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58962025-04-30T06:25:45.226ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
58972025-04-30T06:25:45.227ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58982025-04-30T06:25:45.228ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a93fb06-16c2-4802-a080-8fa59a33619f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
58992025-04-30T06:25:45.228ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
59002025-04-30T06:25:45.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b05c325-952c-46be-9e5c-fbf2eb02b0fe", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59012025-04-30T06:25:45.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
59022025-04-30T06:25:45.236ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59032025-04-30T06:25:45.237ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
59042025-04-30T06:25:45.238ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59052025-04-30T06:25:45.239ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e6c1e5-2dcb-4459-ae7d-f004443332de", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59062025-04-30T06:25:45.239ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
59072025-04-30T06:25:45.248ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "119b1d10-91b3-4d04-8fe5-e877c4d9afb0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59082025-04-30T06:25:45.248ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
59092025-04-30T06:25:45.248ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
59102025-04-30T06:25:45.252ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "be83faad-0ca4-4fe0-85a7-77c60e405697", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59112025-04-30T06:25:45.252ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
59122025-04-30T06:25:45.275ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21a77e65-2510-4ddb-ba72-d845166754ba", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59132025-04-30T06:25:45.275ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
59142025-04-30T06:25:45.282ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dae0feaf-1068-48d0-b416-5205ce127b3f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59152025-04-30T06:25:45.282ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
59162025-04-30T06:25:45.284ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a9f4843-e003-454f-ab71-49301263c356", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59172025-04-30T06:25:45.284ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
59182025-04-30T06:25:45.285ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d03df8d-d0c7-4345-b24f-2f565b80dae9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59192025-04-30T06:25:45.285ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59202025-04-30T06:25:45.285ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59212025-04-30T06:25:45.285ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59222025-04-30T06:25:45.285ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59232025-04-30T06:25:45.286ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59242025-04-30T06:25:45.290ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59252025-04-30T06:25:45.306ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59262025-04-30T06:25:45.306ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59272025-04-30T06:25:45.306ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59282025-04-30T06:25:45.306ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59292025-04-30T06:25:45.306ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59302025-04-30T06:25:45.307ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59312025-04-30T06:25:45.308ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
59322025-04-30T06:25:45.308ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59332025-04-30T06:25:45.309ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 32aeaaba-bd21-4f64-8191-eecb05646644 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59342025-04-30T06:25:45.309ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 32aeaaba-bd21-4f64-8191-eecb05646644 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59352025-04-30T06:25:45.309ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 32aeaaba-bd21-4f64-8191-eecb05646644 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59362025-04-30T06:25:45.310ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 32aeaaba-bd21-4f64-8191-eecb05646644 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59372025-04-30T06:25:45.310ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59382025-04-30T06:25:45.313ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59392025-04-30T06:25:45.314ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59402025-04-30T06:25:45.314ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59412025-04-30T06:25:45.314ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59422025-04-30T06:25:45.333ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 24368 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 32aeaaba-bd21-4f64-8191-eecb05646644 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
59432025-04-30T06:25:45.355ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "32aeaaba-bd21-4f64-8191-eecb05646644", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59442025-04-30T06:25:45.355ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
59452025-04-30T06:25:45.356ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 690d70d8-f4f5-4134-bb99-020d82781979 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
59462025-04-30T06:25:45.356ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 690d70d8-f4f5-4134-bb99-020d82781979 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
59472025-04-30T06:25:45.356ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 690d70d8-f4f5-4134-bb99-020d82781979 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
59482025-04-30T06:25:45.358ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 690d70d8-f4f5-4134-bb99-020d82781979 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
59492025-04-30T06:25:45.360ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
59502025-04-30T06:25:45.373ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 17204 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 690d70d8-f4f5-4134-bb99-020d82781979 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
59512025-04-30T06:25:45.373ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "690d70d8-f4f5-4134-bb99-020d82781979", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:44 GMT"} })
59522025-04-30T06:25:45.373ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59532025-04-30T06:25:45.373ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
59542025-04-30T06:25:45.373ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
59552025-04-30T06:25:45.540ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
59562025-04-30T06:25:45.540ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
59572025-04-30T06:25:45.541ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59582025-04-30T06:25:45.541ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59592025-04-30T06:25:45.541ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59602025-04-30T06:25:45.541ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
59612025-04-30T06:25:45.543ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59622025-04-30T06:25:45.543ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:44498 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
59632025-04-30T06:25:45.543ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59642025-04-30T06:25:45.543ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59652025-04-30T06:25:45.545ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:44498 req_id = b5d0e484-e803-404b-8f21-18a92dfde129 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
59662025-04-30T06:25:45.546ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1833 local_addr = [::1]:42668 method = GET remote_addr = [::1]:44498 req_id = b5d0e484-e803-404b-8f21-18a92dfde129 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
59672025-04-30T06:25:45.549ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d0e484-e803-404b-8f21-18a92dfde129", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
59682025-04-30T06:25:45.615ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
59692025-04-30T06:25:45.615ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
59702025-04-30T06:25:45.615ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59712025-04-30T06:25:45.616ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59722025-04-30T06:25:45.616ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59732025-04-30T06:25:45.616ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
59742025-04-30T06:25:45.617ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59752025-04-30T06:25:45.623ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59762025-04-30T06:25:45.623ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59772025-04-30T06:25:45.623ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
59782025-04-30T06:25:45.623ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59792025-04-30T06:25:45.624ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:49216 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
59802025-04-30T06:25:45.624ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59812025-04-30T06:25:45.624ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
59822025-04-30T06:25:45.645ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
59832025-04-30T06:25:45.646ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
59842025-04-30T06:25:45.646ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:49216 req_id = 78e1b1ed-0508-4e57-b89f-e681c40136b5 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
59852025-04-30T06:25:45.653ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 6399 local_addr = [::1]:47918 method = GET remote_addr = [::1]:49216 req_id = 78e1b1ed-0508-4e57-b89f-e681c40136b5 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
59862025-04-30T06:25:45.653ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59872025-04-30T06:25:45.653ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
59882025-04-30T06:25:45.654ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59892025-04-30T06:25:45.655ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "78e1b1ed-0508-4e57-b89f-e681c40136b5", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
59902025-04-30T06:25:45.772ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
59912025-04-30T06:25:45.772ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
59922025-04-30T06:25:45.774ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:62986 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
59932025-04-30T06:25:45.775ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
59942025-04-30T06:25:45.775ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
59952025-04-30T06:25:45.781ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
59962025-04-30T06:25:45.783ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:62986 req_id = bc69cf4e-3256-43d1-a38d-540f222c2a0c sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
59972025-04-30T06:25:45.783ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 238 local_addr = [::1]:35744 method = GET remote_addr = [::1]:62986 req_id = bc69cf4e-3256-43d1-a38d-540f222c2a0c response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
59982025-04-30T06:25:45.786ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
59992025-04-30T06:25:45.786ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
60002025-04-30T06:25:45.790ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc69cf4e-3256-43d1-a38d-540f222c2a0c", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60012025-04-30T06:25:45.835ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
60022025-04-30T06:25:45.835ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
60032025-04-30T06:25:45.837ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:49181 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
60042025-04-30T06:25:45.837ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:49181 req_id = aa3aecd2-7186-4e15-b59d-a4de3c90d67f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60052025-04-30T06:25:45.840ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2890 local_addr = [::1]:35343 method = GET remote_addr = [::1]:49181 req_id = aa3aecd2-7186-4e15-b59d-a4de3c90d67f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
60062025-04-30T06:25:45.841ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa3aecd2-7186-4e15-b59d-a4de3c90d67f", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60072025-04-30T06:25:45.844ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60082025-04-30T06:25:45.844ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
60092025-04-30T06:25:45.844ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
60102025-04-30T06:25:45.849ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60112025-04-30T06:25:45.849ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60122025-04-30T06:25:45.849ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60132025-04-30T06:25:45.849ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60142025-04-30T06:25:45.850ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60152025-04-30T06:25:45.858ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60162025-04-30T06:25:45.862ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60172025-04-30T06:25:45.862ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60182025-04-30T06:25:45.862ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60192025-04-30T06:25:45.863ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60202025-04-30T06:25:45.888ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
60212025-04-30T06:25:45.899ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
60222025-04-30T06:25:45.900ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
60232025-04-30T06:25:45.900ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.001611495s iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
60242025-04-30T06:25:45.900ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
60252025-04-30T06:25:45.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
60262025-04-30T06:25:45.900ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60272025-04-30T06:25:45.900ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60282025-04-30T06:25:45.900ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60292025-04-30T06:25:45.901ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60302025-04-30T06:25:45.904ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60312025-04-30T06:25:45.909ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
60322025-04-30T06:25:45.909ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
60332025-04-30T06:25:45.909ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
60342025-04-30T06:25:45.910ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
60352025-04-30T06:25:45.914ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
60362025-04-30T06:25:45.914ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
60372025-04-30T06:25:45.915ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b2 42 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .B..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53470 req_id = 6b4d8ac4-97df-41d6-8c04-537229e3bb05
60382025-04-30T06:25:45.915ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45634,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53470 req_id = 6b4d8ac4-97df-41d6-8c04-537229e3bb05
60392025-04-30T06:25:45.915ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
60402025-04-30T06:25:45.915ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
60412025-04-30T06:25:45.915ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60422025-04-30T06:25:45.915ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53470 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6b4d8ac4-97df-41d6-8c04-537229e3bb05
60432025-04-30T06:25:45.916ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3210, tv_nsec: 715225642 } })
60442025-04-30T06:25:45.916ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
60452025-04-30T06:25:45.916ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
60462025-04-30T06:25:45.916ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ba 39 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .9..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55893 req_id = 3d1eddf5-b048-4067-841d-53fa0f2be199
60472025-04-30T06:25:45.916ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47673,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55893 req_id = 3d1eddf5-b048-4067-841d-53fa0f2be199
60482025-04-30T06:25:45.917ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
60492025-04-30T06:25:45.917ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55893 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3d1eddf5-b048-4067-841d-53fa0f2be199
60502025-04-30T06:25:45.917ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cf 6b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .k..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64955 req_id = 6d7b110e-e35b-4bdc-a99b-aad0b8274272
60512025-04-30T06:25:45.917ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53099,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64955 req_id = 6d7b110e-e35b-4bdc-a99b-aad0b8274272
60522025-04-30T06:25:45.917ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60532025-04-30T06:25:45.917ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64955 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6d7b110e-e35b-4bdc-a99b-aad0b8274272
60542025-04-30T06:25:46.026ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60552025-04-30T06:25:46.026ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60562025-04-30T06:25:46.026ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60572025-04-30T06:25:46.026ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60582025-04-30T06:25:46.027ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
60592025-04-30T06:25:46.027ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60602025-04-30T06:25:46.029ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ca f0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61824 req_id = 2d4fced4-f074-4498-8bfd-09a687b61525
60612025-04-30T06:25:46.029ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51952,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61824 req_id = 2d4fced4-f074-4498-8bfd-09a687b61525
60622025-04-30T06:25:46.029ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
60632025-04-30T06:25:46.029ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 51952, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61824 req_id = 2d4fced4-f074-4498-8bfd-09a687b61525
60642025-04-30T06:25:46.031ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
60652025-04-30T06:25:46.031ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
60662025-04-30T06:25:46.031ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
60672025-04-30T06:25:46.032ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60682025-04-30T06:25:46.045ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60692025-04-30T06:25:46.045ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60702025-04-30T06:25:46.045ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
60712025-04-30T06:25:46.045ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60722025-04-30T06:25:46.048ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "29dfb1e2-0683-411e-bf02-4435308f51e7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60732025-04-30T06:25:46.048ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
60742025-04-30T06:25:46.050ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2eec773-306a-4766-878f-ac23a0e157e3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60752025-04-30T06:25:46.050ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
60762025-04-30T06:25:46.057ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60772025-04-30T06:25:46.057ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
60782025-04-30T06:25:46.058ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60792025-04-30T06:25:46.061ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcef6bd9-dc8e-4e68-bcd2-3d02632c4a53", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60802025-04-30T06:25:46.062ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
60812025-04-30T06:25:46.066ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a0e5f77-198b-4adb-970a-7b1c8f5f3ab5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60822025-04-30T06:25:46.067ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
60832025-04-30T06:25:46.076ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12c84e35-982f-4750-bbbc-10d592574908", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60842025-04-30T06:25:46.077ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
60852025-04-30T06:25:46.079ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d1ab336-6676-43cf-8f7d-7f8711190572", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60862025-04-30T06:25:46.079ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
60872025-04-30T06:25:46.083ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2554896a-422b-4190-b973-0c67ea14a2eb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60882025-04-30T06:25:46.083ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
60892025-04-30T06:25:46.086ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59f43cd9-3fb9-4d40-aca2-1fb29e6e111d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60902025-04-30T06:25:46.086ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
60912025-04-30T06:25:46.087ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0486b14a-8a11-418b-bdaf-481931cfe60c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60922025-04-30T06:25:46.088ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
60932025-04-30T06:25:46.091ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cb2d628-70fa-471f-8fed-00495a216eed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60942025-04-30T06:25:46.091ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
60952025-04-30T06:25:46.093ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51b6e697-ddc0-4a82-b88d-f78e784a8e39", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60962025-04-30T06:25:46.094ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
60972025-04-30T06:25:46.096ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a004834f-664a-4da9-95fe-ef263ec4e2a1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
60982025-04-30T06:25:46.096ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
60992025-04-30T06:25:46.097ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
61002025-04-30T06:25:46.098ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b270a2e-d515-4e74-8811-567850cceeaf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61012025-04-30T06:25:46.099ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
61022025-04-30T06:25:46.100ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f26734e-54f1-4644-8940-bad4b1b945a9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61032025-04-30T06:25:46.101ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
61042025-04-30T06:25:46.102ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e7b135a-e0b7-44b3-a6be-4cf4005e5e10", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61052025-04-30T06:25:46.102ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
61062025-04-30T06:25:46.104ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee81e2f5-de98-49b1-b186-d987030713cc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61072025-04-30T06:25:46.105ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
61082025-04-30T06:25:46.107ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70bbb2e9-64b3-48b8-b6f8-17f9e2f5e034", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61092025-04-30T06:25:46.107ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
61102025-04-30T06:25:46.109ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6341280a-2e03-42ba-b9c9-a892ce7c9438", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61112025-04-30T06:25:46.109ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
61122025-04-30T06:25:46.111ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ef40f9c-72b4-4ca6-9ee0-c8fd35e32ef6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61132025-04-30T06:25:46.111ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
61142025-04-30T06:25:46.112ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1bcd76e-ad27-4c4c-9790-bfc2937c66b3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61152025-04-30T06:25:46.113ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
61162025-04-30T06:25:46.115ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4a22a3d-d2ac-45fe-a87a-571e61a34964", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61172025-04-30T06:25:46.115ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
61182025-04-30T06:25:46.116ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a68c638-e5b0-43ba-a426-263efb31a988", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61192025-04-30T06:25:46.117ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
61202025-04-30T06:25:46.118ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9c2107d-a758-427f-ae92-c4b39b846b63", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61212025-04-30T06:25:46.118ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
61222025-04-30T06:25:46.120ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd6bf2fd-4e5e-4df6-90c3-c6bcc8ef2a50", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61232025-04-30T06:25:46.120ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
61242025-04-30T06:25:46.122ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59960314-a37b-4477-b0c9-50746af13a2f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61252025-04-30T06:25:46.122ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
61262025-04-30T06:25:46.124ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acf1322f-6c6b-4592-bd9a-61a203e34bee", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61272025-04-30T06:25:46.124ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
61282025-04-30T06:25:46.127ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2057f32d-83ec-46c1-9096-550a327ed3f8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61292025-04-30T06:25:46.127ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
61302025-04-30T06:25:46.129ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8df2232-bf9b-43ba-ab1a-d050dc368bcf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61312025-04-30T06:25:46.129ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
61322025-04-30T06:25:46.130ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3928d9-9519-40c7-931c-caa332b2e3fb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61332025-04-30T06:25:46.131ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
61342025-04-30T06:25:46.132ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d98be67e-96c4-43f0-ae32-4bc3a1876f04", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61352025-04-30T06:25:46.132ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
61362025-04-30T06:25:46.133ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "948d180e-a2b1-461f-a669-02c4ca0e6772", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61372025-04-30T06:25:46.133ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
61382025-04-30T06:25:46.134ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78618aea-fd68-4f4b-99cf-ab13062bb7ce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61392025-04-30T06:25:46.134ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
61402025-04-30T06:25:46.135ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71162d71-d16c-45b9-a781-e39056aad632", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61412025-04-30T06:25:46.136ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
61422025-04-30T06:25:46.137ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e094d47-e9ae-4c5e-a5d9-957c1f3ff125", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61432025-04-30T06:25:46.137ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
61442025-04-30T06:25:46.139ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5779bb37-5535-4dad-aac9-95286b5a5702", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61452025-04-30T06:25:46.139ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
61462025-04-30T06:25:46.142ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c8d1b3-2142-4d86-a08c-53048d3f9952", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61472025-04-30T06:25:46.143ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
61482025-04-30T06:25:46.152ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61492025-04-30T06:25:46.152ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61502025-04-30T06:25:46.152ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61512025-04-30T06:25:46.152ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
61522025-04-30T06:25:46.153ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61532025-04-30T06:25:46.170ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61542025-04-30T06:25:46.177ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61552025-04-30T06:25:46.177ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61562025-04-30T06:25:46.177ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
61572025-04-30T06:25:46.178ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61582025-04-30T06:25:46.183ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61592025-04-30T06:25:46.183ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
61602025-04-30T06:25:46.184ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61612025-04-30T06:25:46.187ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91ed6cce-69f5-4d2d-b19a-40b1082fa000", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61622025-04-30T06:25:46.187ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
61632025-04-30T06:25:46.197ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fda5c234-bb3c-42f3-9268-2321a8917ba0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61642025-04-30T06:25:46.197ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
61652025-04-30T06:25:46.201ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d32ad4c-287c-47fd-b40d-986005263117", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61662025-04-30T06:25:46.201ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
61672025-04-30T06:25:46.204ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e00cd8c-adcc-4a08-b7f7-d28f05e7afc8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61682025-04-30T06:25:46.204ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
61692025-04-30T06:25:46.208ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1604997f-d6b9-47cf-83d5-7580f4efb197", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61702025-04-30T06:25:46.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
61712025-04-30T06:25:46.208ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "45e4007b-5a46-4454-952e-a11bccb1b186", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61722025-04-30T06:25:46.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
61732025-04-30T06:25:46.211ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2792d72d-18a4-473a-abf5-93e7b90aede5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61742025-04-30T06:25:46.211ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
61752025-04-30T06:25:46.213ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87755e04-8907-45d3-872a-fecf749ef5c3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61762025-04-30T06:25:46.213ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
61772025-04-30T06:25:46.214ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3fc57d-fe5d-485b-84cb-f65a7543cf39", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61782025-04-30T06:25:46.214ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
61792025-04-30T06:25:46.214ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
61802025-04-30T06:25:46.220ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "da183249-880e-4a6b-846e-ac6d9a83d2ea", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61812025-04-30T06:25:46.221ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
61822025-04-30T06:25:46.222ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
61832025-04-30T06:25:46.225ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c184a1b4-9259-4ffb-90f9-60179088bc50", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61842025-04-30T06:25:46.225ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
61852025-04-30T06:25:46.227ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "429a23e7-f110-46ab-832c-8f6b89402dc6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61862025-04-30T06:25:46.227ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
61872025-04-30T06:25:46.229ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffa006fe-bf9b-44a2-9ae1-2380e48fe25c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61882025-04-30T06:25:46.229ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
61892025-04-30T06:25:46.231ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69affa20-929b-4b28-9d44-6421ddde06d3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
61902025-04-30T06:25:46.231ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61912025-04-30T06:25:46.231ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61922025-04-30T06:25:46.231ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61932025-04-30T06:25:46.231ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61942025-04-30T06:25:46.232ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61952025-04-30T06:25:46.235ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61962025-04-30T06:25:46.239ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61972025-04-30T06:25:46.240ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61982025-04-30T06:25:46.240ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61992025-04-30T06:25:46.240ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62002025-04-30T06:25:46.240ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62012025-04-30T06:25:46.240ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62022025-04-30T06:25:46.243ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62032025-04-30T06:25:46.250ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
62042025-04-30T06:25:46.250ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62052025-04-30T06:25:46.250ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62062025-04-30T06:25:46.252ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62072025-04-30T06:25:46.388ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
62082025-04-30T06:25:46.388ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
62092025-04-30T06:25:46.388ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62102025-04-30T06:25:46.388ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62112025-04-30T06:25:46.388ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62122025-04-30T06:25:46.388ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
62132025-04-30T06:25:46.389ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62142025-04-30T06:25:46.389ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:41922 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
62152025-04-30T06:25:46.422ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:41922 req_id = 747bc182-f8af-498f-a230-db2865a8fa9e sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
62162025-04-30T06:25:46.422ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62172025-04-30T06:25:46.423ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62182025-04-30T06:25:46.423ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 589 local_addr = [::1]:42668 method = GET remote_addr = [::1]:41922 req_id = 747bc182-f8af-498f-a230-db2865a8fa9e response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
62192025-04-30T06:25:46.424ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "747bc182-f8af-498f-a230-db2865a8fa9e", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:45 GMT"} })
62202025-04-30T06:25:46.526ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
62212025-04-30T06:25:46.526ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
62222025-04-30T06:25:46.529ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62232025-04-30T06:25:46.536ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62242025-04-30T06:25:46.536ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62252025-04-30T06:25:46.536ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
62262025-04-30T06:25:46.537ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62272025-04-30T06:25:46.537ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:50519 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
62282025-04-30T06:25:46.550ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62292025-04-30T06:25:46.551ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62302025-04-30T06:25:46.551ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62312025-04-30T06:25:46.551ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
62322025-04-30T06:25:46.554ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:50519 req_id = 69e8740a-4d5d-4a8a-ac1a-e421caefde82 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
62332025-04-30T06:25:46.556ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1928 local_addr = [::1]:47918 method = GET remote_addr = [::1]:50519 req_id = 69e8740a-4d5d-4a8a-ac1a-e421caefde82 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
62342025-04-30T06:25:46.558ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e8740a-4d5d-4a8a-ac1a-e421caefde82", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
62352025-04-30T06:25:46.633ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
62362025-04-30T06:25:46.634ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
62372025-04-30T06:25:46.636ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62382025-04-30T06:25:46.637ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
62392025-04-30T06:25:46.638ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62402025-04-30T06:25:46.651ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:54021 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
62412025-04-30T06:25:46.652ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62422025-04-30T06:25:46.652ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62432025-04-30T06:25:46.656ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
62442025-04-30T06:25:46.656ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
62452025-04-30T06:25:46.660ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:54021 req_id = d8186612-c94c-444d-86b8-3b2179cea9f0 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
62462025-04-30T06:25:46.670ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 10104 local_addr = [::1]:35744 method = GET remote_addr = [::1]:54021 req_id = d8186612-c94c-444d-86b8-3b2179cea9f0 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
62472025-04-30T06:25:46.671ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
62482025-04-30T06:25:46.672ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
62492025-04-30T06:25:46.673ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8186612-c94c-444d-86b8-3b2179cea9f0", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
62502025-04-30T06:25:46.729ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
62512025-04-30T06:25:46.729ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
62522025-04-30T06:25:46.730ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:32998 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
62532025-04-30T06:25:46.737ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
62542025-04-30T06:25:46.738ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
62552025-04-30T06:25:46.738ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:32998 req_id = fc668cfd-55c4-4e15-99b4-1ba8f30876e3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62562025-04-30T06:25:46.738ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 703 local_addr = [::1]:35343 method = GET remote_addr = [::1]:32998 req_id = fc668cfd-55c4-4e15-99b4-1ba8f30876e3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
62572025-04-30T06:25:46.742ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc668cfd-55c4-4e15-99b4-1ba8f30876e3", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
62582025-04-30T06:25:46.744ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62592025-04-30T06:25:46.744ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
62602025-04-30T06:25:46.744ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
62612025-04-30T06:25:46.753ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62622025-04-30T06:25:46.753ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62632025-04-30T06:25:46.753ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62642025-04-30T06:25:46.753ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62652025-04-30T06:25:46.754ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62662025-04-30T06:25:46.756ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
62672025-04-30T06:25:46.759ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62682025-04-30T06:25:46.764ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62692025-04-30T06:25:46.764ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62702025-04-30T06:25:46.764ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62712025-04-30T06:25:46.764ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62722025-04-30T06:25:46.810ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
62732025-04-30T06:25:46.820ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62742025-04-30T06:25:46.821ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62752025-04-30T06:25:46.821ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 920.86849ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62762025-04-30T06:25:46.821ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
62772025-04-30T06:25:46.821ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62782025-04-30T06:25:46.821ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62792025-04-30T06:25:46.821ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62802025-04-30T06:25:46.821ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62812025-04-30T06:25:46.822ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62822025-04-30T06:25:46.824ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62832025-04-30T06:25:46.828ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62842025-04-30T06:25:46.828ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62852025-04-30T06:25:46.828ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62862025-04-30T06:25:46.829ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62872025-04-30T06:25:46.832ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
62882025-04-30T06:25:46.832ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62892025-04-30T06:25:46.833ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: dd 58 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .X..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63119 req_id = 50a59d9f-832f-42ae-b935-da94b5138346
62902025-04-30T06:25:46.833ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56664,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63119 req_id = 50a59d9f-832f-42ae-b935-da94b5138346
62912025-04-30T06:25:46.833ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
62922025-04-30T06:25:46.833ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
62932025-04-30T06:25:46.834ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62942025-04-30T06:25:46.834ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63119 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 50a59d9f-832f-42ae-b935-da94b5138346
62952025-04-30T06:25:46.834ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3211, tv_nsec: 633474362 } })
62962025-04-30T06:25:46.834ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
62972025-04-30T06:25:46.834ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
62982025-04-30T06:25:46.835ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cb 81 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53178 req_id = 2aa116c5-b516-4b90-a032-fac23fd8c257
62992025-04-30T06:25:46.835ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52097,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53178 req_id = 2aa116c5-b516-4b90-a032-fac23fd8c257
63002025-04-30T06:25:46.835ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
63012025-04-30T06:25:46.835ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53178 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2aa116c5-b516-4b90-a032-fac23fd8c257
63022025-04-30T06:25:46.835ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7e b8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ~...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57086 req_id = ef03fabb-3684-4a1f-aaf4-730c79551217
63032025-04-30T06:25:46.835ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32440,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57086 req_id = ef03fabb-3684-4a1f-aaf4-730c79551217
63042025-04-30T06:25:46.835ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
63052025-04-30T06:25:46.835ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57086 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ef03fabb-3684-4a1f-aaf4-730c79551217
63062025-04-30T06:25:46.959ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
63072025-04-30T06:25:46.959ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63082025-04-30T06:25:46.959ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63092025-04-30T06:25:46.959ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63102025-04-30T06:25:46.959ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
63112025-04-30T06:25:46.960ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63122025-04-30T06:25:46.961ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d3 6f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .o..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54710 req_id = f0208b9f-fb3c-466f-89be-89beb18a0162
63132025-04-30T06:25:46.961ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54127,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54710 req_id = f0208b9f-fb3c-466f-89be-89beb18a0162
63142025-04-30T06:25:46.961ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
63152025-04-30T06:25:46.962ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54127, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54710 req_id = f0208b9f-fb3c-466f-89be-89beb18a0162
63162025-04-30T06:25:46.962ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
63172025-04-30T06:25:46.962ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
63182025-04-30T06:25:46.962ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
63192025-04-30T06:25:46.963ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63202025-04-30T06:25:46.970ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63212025-04-30T06:25:46.970ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63222025-04-30T06:25:46.970ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
63232025-04-30T06:25:46.971ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63242025-04-30T06:25:46.972ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9826c63c-7ddd-451e-880e-69921b7aca22", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63252025-04-30T06:25:46.973ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
63262025-04-30T06:25:46.975ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63272025-04-30T06:25:46.975ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
63282025-04-30T06:25:46.976ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63292025-04-30T06:25:46.977ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e04132e-99bc-4a55-bfa3-5687ed702d1d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63302025-04-30T06:25:46.977ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
63312025-04-30T06:25:46.980ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9879e7ce-56de-49d4-baa3-ec17da002575", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63322025-04-30T06:25:46.980ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
63332025-04-30T06:25:46.984ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15efbc56-3305-4a13-a9bd-6a9ac407ab69", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63342025-04-30T06:25:46.985ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
63352025-04-30T06:25:46.987ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6094be08-a001-4c84-9b67-ad91d873b32d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63362025-04-30T06:25:46.987ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
63372025-04-30T06:25:46.989ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6742cfea-24ea-4509-af39-db02f49bdb7d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63382025-04-30T06:25:46.989ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
63392025-04-30T06:25:46.992ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "baeef72d-300b-4411-8953-1eb82141dbe0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63402025-04-30T06:25:46.992ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
63412025-04-30T06:25:46.996ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3f7dca0-c1f5-4cfb-96e9-4d40bbd0ec81", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63422025-04-30T06:25:46.996ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
63432025-04-30T06:25:47.000ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71a1e49f-b32d-4a5c-a520-c9c1fb8cb67f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63442025-04-30T06:25:47.001ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
63452025-04-30T06:25:47.003ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df86e32a-2705-47c5-ada1-58d4e8a3f9c1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63462025-04-30T06:25:47.003ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
63472025-04-30T06:25:47.006ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07679136-4d7b-4634-8e75-f4883426fdbf", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63482025-04-30T06:25:47.006ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
63492025-04-30T06:25:47.008ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d2ab739-3e33-47c9-9d29-8edccb74dfaa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63502025-04-30T06:25:47.008ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
63512025-04-30T06:25:47.010ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93ad1111-4c62-44d6-bb5f-799aeb1207ac", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63522025-04-30T06:25:47.010ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
63532025-04-30T06:25:47.015ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
63542025-04-30T06:25:47.016ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc776964-eb16-4374-b871-c1099c1f014c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63552025-04-30T06:25:47.017ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
63562025-04-30T06:25:47.019ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c726d86f-72a3-4075-9373-cd7ac45859ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63572025-04-30T06:25:47.020ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
63582025-04-30T06:25:47.023ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11c64621-9659-4dde-b313-3f1105c06641", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63592025-04-30T06:25:47.023ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
63602025-04-30T06:25:47.025ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e3f2bb-e4ad-41ef-8586-3db56a065de7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63612025-04-30T06:25:47.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
63622025-04-30T06:25:47.031ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9228ce2-76bb-41ca-8c37-375c48d511f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63632025-04-30T06:25:47.031ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
63642025-04-30T06:25:47.034ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "193366c6-1d42-407f-9acd-8be2861336ad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63652025-04-30T06:25:47.034ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
63662025-04-30T06:25:47.044ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba156011-f557-4ba3-baca-91c44bd8ca24", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63672025-04-30T06:25:47.045ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
63682025-04-30T06:25:47.047ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "82cb6fd7-274a-490f-a989-d5d6543b7c10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63692025-04-30T06:25:47.047ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
63702025-04-30T06:25:47.050ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4d97601-06b6-464a-b515-d06f5c5ce2a4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63712025-04-30T06:25:47.050ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
63722025-04-30T06:25:47.053ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "967af3c2-3cf8-40f7-8471-275dff8a8349", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63732025-04-30T06:25:47.053ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
63742025-04-30T06:25:47.055ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56e8c68b-afe9-4c0b-9879-d39c2666e2bc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63752025-04-30T06:25:47.055ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
63762025-04-30T06:25:47.057ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0f22091-cd0c-412c-a660-a87c118bd2c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63772025-04-30T06:25:47.057ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
63782025-04-30T06:25:47.059ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03dbba0a-139a-4280-b3a3-4bef6f78a702", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63792025-04-30T06:25:47.059ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
63802025-04-30T06:25:47.061ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eae1ba3-1834-448b-aafc-071b58918b55", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63812025-04-30T06:25:47.061ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
63822025-04-30T06:25:47.063ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "637b7995-61aa-40b0-861d-ad417f837a8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63832025-04-30T06:25:47.063ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
63842025-04-30T06:25:47.065ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "566619c4-4f2b-4ba3-b874-f1bb78246df2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63852025-04-30T06:25:47.065ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
63862025-04-30T06:25:47.066ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63872025-04-30T06:25:47.066ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63882025-04-30T06:25:47.066ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63892025-04-30T06:25:47.066ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
63902025-04-30T06:25:47.067ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63912025-04-30T06:25:47.070ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b86b9c0-449e-4b1b-922d-f0ac18af65f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63922025-04-30T06:25:47.070ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
63932025-04-30T06:25:47.070ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63942025-04-30T06:25:47.075ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63952025-04-30T06:25:47.075ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63962025-04-30T06:25:47.075ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
63972025-04-30T06:25:47.076ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63982025-04-30T06:25:47.077ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa988882-c30e-4b77-8ca7-0d3420d63eca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
63992025-04-30T06:25:47.077ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
64002025-04-30T06:25:47.079ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64012025-04-30T06:25:47.079ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
64022025-04-30T06:25:47.080ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64032025-04-30T06:25:47.081ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50921da4-1f83-446c-becf-c361c73b9ffa", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64042025-04-30T06:25:47.081ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
64052025-04-30T06:25:47.085ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "220f738e-df5f-432e-82f2-4dcb9b0366b3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64062025-04-30T06:25:47.085ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
64072025-04-30T06:25:47.086ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aefe7bf-1904-4b00-b788-9db18f277b55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64082025-04-30T06:25:47.086ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
64092025-04-30T06:25:47.088ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57b66827-6238-4992-bc73-1cce15afab5e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64102025-04-30T06:25:47.089ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
64112025-04-30T06:25:47.094ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdd80922-a826-45cb-92cb-d9866570128a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64122025-04-30T06:25:47.094ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
64132025-04-30T06:25:47.096ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3fcca9-027c-48f8-8309-800ffbafc435", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64142025-04-30T06:25:47.097ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
64152025-04-30T06:25:47.100ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30df1aad-d19b-490d-869b-39e9d3b05fe4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64162025-04-30T06:25:47.100ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
64172025-04-30T06:25:47.105ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c09df2-e989-4ea4-bdcf-8a7ccac93e52", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64182025-04-30T06:25:47.105ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
64192025-04-30T06:25:47.116ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f93e468-e46e-4aa7-8905-914a9ec6c55c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64202025-04-30T06:25:47.116ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
64212025-04-30T06:25:47.117ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
64222025-04-30T06:25:47.121ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e6114f6-bfa1-4c98-abf2-9277d14de214", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64232025-04-30T06:25:47.121ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
64242025-04-30T06:25:47.123ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c30351a-d7a3-46de-835d-621a61682768", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64252025-04-30T06:25:47.123ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
64262025-04-30T06:25:47.124ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19de887e-209f-4e22-aa8b-8a9fdfb3ec36", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64272025-04-30T06:25:47.124ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
64282025-04-30T06:25:47.125ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "873c173a-e43b-40d5-95f5-1fd3ed1e7817", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64292025-04-30T06:25:47.125ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
64302025-04-30T06:25:47.128ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "32b2ce7d-717c-4985-8071-9082bae87897", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64312025-04-30T06:25:47.128ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
64322025-04-30T06:25:47.128ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
64332025-04-30T06:25:47.132ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "258eb2c4-f460-43b9-9a63-1c0597ae8fc6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64342025-04-30T06:25:47.133ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
64352025-04-30T06:25:47.137ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b0206b0-3ea4-44aa-8b58-9c30701fabf5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64362025-04-30T06:25:47.137ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
64372025-04-30T06:25:47.160ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1fd3a22-94a8-4b84-9a09-dcb49cb0a838", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64382025-04-30T06:25:47.160ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
64392025-04-30T06:25:47.168ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64402025-04-30T06:25:47.168ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64412025-04-30T06:25:47.168ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64422025-04-30T06:25:47.168ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
64432025-04-30T06:25:47.169ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64442025-04-30T06:25:47.174ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf97b3ea-66c6-4d3b-b080-c32e355e1b55", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64452025-04-30T06:25:47.174ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
64462025-04-30T06:25:47.174ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64472025-04-30T06:25:47.181ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64482025-04-30T06:25:47.181ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64492025-04-30T06:25:47.181ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
64502025-04-30T06:25:47.182ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64512025-04-30T06:25:47.189ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64522025-04-30T06:25:47.189ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
64532025-04-30T06:25:47.190ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64542025-04-30T06:25:47.192ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c2bc7b5-b7fa-4046-b5c4-d3edb392248f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64552025-04-30T06:25:47.192ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64562025-04-30T06:25:47.192ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64572025-04-30T06:25:47.192ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64582025-04-30T06:25:47.193ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64592025-04-30T06:25:47.193ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64602025-04-30T06:25:47.200ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64612025-04-30T06:25:47.204ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64622025-04-30T06:25:47.204ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
64632025-04-30T06:25:47.204ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64642025-04-30T06:25:47.204ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64652025-04-30T06:25:47.204ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64662025-04-30T06:25:47.205ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64672025-04-30T06:25:47.208ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64682025-04-30T06:25:47.214ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
64692025-04-30T06:25:47.215ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64702025-04-30T06:25:47.215ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64712025-04-30T06:25:47.215ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64722025-04-30T06:25:47.263ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
64732025-04-30T06:25:47.263ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
64742025-04-30T06:25:47.264ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:63058 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
64752025-04-30T06:25:47.265ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:63058 req_id = 955a3883-f47c-4511-8062-84ce94a3819b sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
64762025-04-30T06:25:47.265ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 470 local_addr = [::1]:42668 method = GET remote_addr = [::1]:63058 req_id = 955a3883-f47c-4511-8062-84ce94a3819b response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
64772025-04-30T06:25:47.266ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "955a3883-f47c-4511-8062-84ce94a3819b", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64782025-04-30T06:25:47.355ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
64792025-04-30T06:25:47.356ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
64802025-04-30T06:25:47.372ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:61552 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
64812025-04-30T06:25:47.373ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
64822025-04-30T06:25:47.374ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
64832025-04-30T06:25:47.374ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 18f23f93-8794-45fb-be85-26056e73be7a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
64842025-04-30T06:25:47.375ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:61552 req_id = 59c10537-0776-459a-9bd1-2b9d952765f5 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
64852025-04-30T06:25:47.375ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 18f23f93-8794-45fb-be85-26056e73be7a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
64862025-04-30T06:25:47.376ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 18f23f93-8794-45fb-be85-26056e73be7a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
64872025-04-30T06:25:47.378ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 18f23f93-8794-45fb-be85-26056e73be7a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
64882025-04-30T06:25:47.378ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2940 local_addr = [::1]:47918 method = GET remote_addr = [::1]:61552 req_id = 59c10537-0776-459a-9bd1-2b9d952765f5 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
64892025-04-30T06:25:47.380ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c10537-0776-459a-9bd1-2b9d952765f5", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64902025-04-30T06:25:47.441ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
64912025-04-30T06:25:47.441ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
64922025-04-30T06:25:47.442ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:57815 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
64932025-04-30T06:25:47.443ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64942025-04-30T06:25:47.443ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
64952025-04-30T06:25:47.444ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 69137 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 18f23f93-8794-45fb-be85-26056e73be7a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
64962025-04-30T06:25:47.446ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:57815 req_id = 8d296d10-02b5-44e6-9dcb-d0a3e5e29821 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
64972025-04-30T06:25:47.446ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "18f23f93-8794-45fb-be85-26056e73be7a", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
64982025-04-30T06:25:47.446ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
64992025-04-30T06:25:47.450ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3910 local_addr = [::1]:35744 method = GET remote_addr = [::1]:57815 req_id = 8d296d10-02b5-44e6-9dcb-d0a3e5e29821 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
65002025-04-30T06:25:47.451ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = e42d67b3-a818-4e2c-a13b-f5488192b118 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
65012025-04-30T06:25:47.451ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = e42d67b3-a818-4e2c-a13b-f5488192b118 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
65022025-04-30T06:25:47.452ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = e42d67b3-a818-4e2c-a13b-f5488192b118 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
65032025-04-30T06:25:47.454ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = e42d67b3-a818-4e2c-a13b-f5488192b118 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
65042025-04-30T06:25:47.454ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d296d10-02b5-44e6-9dcb-d0a3e5e29821", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
65052025-04-30T06:25:47.526ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
65062025-04-30T06:25:47.526ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
65072025-04-30T06:25:47.527ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65082025-04-30T06:25:47.529ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65092025-04-30T06:25:47.529ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65102025-04-30T06:25:47.529ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65112025-04-30T06:25:47.537ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:34203 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
65122025-04-30T06:25:47.538ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
65132025-04-30T06:25:47.541ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:34203 req_id = 28a41af9-7ddf-4fc0-903a-6e67acb0366b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65142025-04-30T06:25:47.550ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 9425 local_addr = [::1]:35343 method = GET remote_addr = [::1]:34203 req_id = 28a41af9-7ddf-4fc0-903a-6e67acb0366b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65152025-04-30T06:25:47.551ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 100338 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = e42d67b3-a818-4e2c-a13b-f5488192b118 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
65162025-04-30T06:25:47.555ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "e42d67b3-a818-4e2c-a13b-f5488192b118", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
65172025-04-30T06:25:47.556ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
65182025-04-30T06:25:47.556ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
65192025-04-30T06:25:47.556ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
65202025-04-30T06:25:47.556ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "28a41af9-7ddf-4fc0-903a-6e67acb0366b", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:46 GMT"} })
65212025-04-30T06:25:47.564ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65222025-04-30T06:25:47.564ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65232025-04-30T06:25:47.564ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
65242025-04-30T06:25:47.574ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65252025-04-30T06:25:47.574ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65262025-04-30T06:25:47.574ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65272025-04-30T06:25:47.574ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65282025-04-30T06:25:47.574ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65292025-04-30T06:25:47.588ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65302025-04-30T06:25:47.599ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
65312025-04-30T06:25:47.600ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65322025-04-30T06:25:47.600ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65332025-04-30T06:25:47.600ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65342025-04-30T06:25:47.601ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65352025-04-30T06:25:47.601ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65362025-04-30T06:25:47.601ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65372025-04-30T06:25:47.601ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
65382025-04-30T06:25:47.601ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65392025-04-30T06:25:47.610ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
65402025-04-30T06:25:47.610ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
65412025-04-30T06:25:47.611ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
65422025-04-30T06:25:47.613ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
65432025-04-30T06:25:47.613ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65442025-04-30T06:25:47.622ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65452025-04-30T06:25:47.622ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65462025-04-30T06:25:47.622ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
65472025-04-30T06:25:47.622ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65482025-04-30T06:25:47.634ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
65492025-04-30T06:25:47.634ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
65502025-04-30T06:25:47.644ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
65512025-04-30T06:25:47.644ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
65522025-04-30T06:25:47.705ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
65532025-04-30T06:25:47.705ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
65542025-04-30T06:25:47.706ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 884.827141ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
65552025-04-30T06:25:47.706ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
65562025-04-30T06:25:47.706ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65572025-04-30T06:25:47.706ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65582025-04-30T06:25:47.706ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65592025-04-30T06:25:47.706ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65602025-04-30T06:25:47.707ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65612025-04-30T06:25:47.710ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65622025-04-30T06:25:47.718ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
65632025-04-30T06:25:47.718ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65642025-04-30T06:25:47.718ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65652025-04-30T06:25:47.719ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65662025-04-30T06:25:47.720ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65672025-04-30T06:25:47.720ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
65682025-04-30T06:25:47.724ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65692025-04-30T06:25:47.727ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
65702025-04-30T06:25:47.729ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
65712025-04-30T06:25:47.730ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
65722025-04-30T06:25:47.732ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 77 0c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 w............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51137 req_id = 11e962bc-da22-40fc-8ecb-ffb2368d3144
65732025-04-30T06:25:47.732ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30476,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51137 req_id = 11e962bc-da22-40fc-8ecb-ffb2368d3144
65742025-04-30T06:25:47.733ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
65752025-04-30T06:25:47.733ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
65762025-04-30T06:25:47.752ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
65772025-04-30T06:25:47.753ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51137 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 11e962bc-da22-40fc-8ecb-ffb2368d3144
65782025-04-30T06:25:47.755ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3212, tv_nsec: 553882516 } })
65792025-04-30T06:25:47.755ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
65802025-04-30T06:25:47.755ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
65812025-04-30T06:25:47.755ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 24 35 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e $5..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59569 req_id = b5f42627-6bf8-4b42-9d7b-6a756c5eec76
65822025-04-30T06:25:47.755ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9269,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59569 req_id = b5f42627-6bf8-4b42-9d7b-6a756c5eec76
65832025-04-30T06:25:47.756ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
65842025-04-30T06:25:47.756ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59569 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b5f42627-6bf8-4b42-9d7b-6a756c5eec76
65852025-04-30T06:25:47.756ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 34 31 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 41..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51621 req_id = 095c71e4-eae4-4ac6-b075-a08d8f9d3f1f
65862025-04-30T06:25:47.756ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51621 req_id = 095c71e4-eae4-4ac6-b075-a08d8f9d3f1f
65872025-04-30T06:25:47.756ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
65882025-04-30T06:25:47.756ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51621 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 095c71e4-eae4-4ac6-b075-a08d8f9d3f1f
65892025-04-30T06:25:47.878ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
65902025-04-30T06:25:47.879ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: bc 64 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .d..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53061 req_id = 0e00a432-ad3e-45f5-b1ce-343f1e4cc1dc
65912025-04-30T06:25:47.879ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48228,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53061 req_id = 0e00a432-ad3e-45f5-b1ce-343f1e4cc1dc
65922025-04-30T06:25:47.879ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
65932025-04-30T06:25:47.880ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 48228, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53061 req_id = 0e00a432-ad3e-45f5-b1ce-343f1e4cc1dc
65942025-04-30T06:25:47.880ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
65952025-04-30T06:25:47.880ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
65962025-04-30T06:25:47.880ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
65972025-04-30T06:25:47.883ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c65a86e1-c50e-42f8-ac4d-86acb2b8d724", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
65982025-04-30T06:25:47.883ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
65992025-04-30T06:25:47.887ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf75bbd3-3bb3-4b37-b95e-970a37e03daf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66002025-04-30T06:25:47.887ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
66012025-04-30T06:25:47.889ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a310d567-ad14-41b3-b83f-480e47aecab6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66022025-04-30T06:25:47.890ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
66032025-04-30T06:25:47.891ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d363f3a-1886-48ee-b0b9-59a091409ae2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66042025-04-30T06:25:47.892ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
66052025-04-30T06:25:47.894ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41d3c1af-b9a9-49b4-89f6-f37e208e739b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66062025-04-30T06:25:47.894ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
66072025-04-30T06:25:47.896ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce9453e7-9f9d-4369-892b-ac6f41904247", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66082025-04-30T06:25:47.897ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
66092025-04-30T06:25:47.899ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ce8ec8-3952-4692-bf44-8f36d3d431b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66102025-04-30T06:25:47.899ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
66112025-04-30T06:25:47.902ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bee1862-26bf-4b6a-b546-02b172af95b4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66122025-04-30T06:25:47.902ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
66132025-04-30T06:25:47.903ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b155989-f4ac-4d89-99a4-cc8b069bafd8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66142025-04-30T06:25:47.903ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
66152025-04-30T06:25:47.907ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54255202-a63d-46d8-b95d-32a084194e5c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66162025-04-30T06:25:47.907ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
66172025-04-30T06:25:47.908ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
66182025-04-30T06:25:47.911ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "be018055-0247-42ca-b7a5-974144b25b3b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66192025-04-30T06:25:47.911ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
66202025-04-30T06:25:47.912ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d281ecd-aacd-4d45-a3bc-295069311185", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66212025-04-30T06:25:47.913ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
66222025-04-30T06:25:47.915ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90f02d49-61e7-4137-a005-4180e7c83062", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66232025-04-30T06:25:47.915ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
66242025-04-30T06:25:47.919ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6ce840d-f11e-4a8e-a666-0232517a30c7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66252025-04-30T06:25:47.919ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
66262025-04-30T06:25:47.923ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "658eafeb-bb6a-4dbd-81b4-1985a19677d4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66272025-04-30T06:25:47.923ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
66282025-04-30T06:25:47.927ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50cbaf55-5834-4c54-a854-89b9014cd065", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66292025-04-30T06:25:47.927ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
66302025-04-30T06:25:47.929ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43963aba-422f-401d-a809-b4e3abe452e7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66312025-04-30T06:25:47.929ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
66322025-04-30T06:25:47.932ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98c51f12-9215-4018-a1e9-eb7cbff854fd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66332025-04-30T06:25:47.932ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
66342025-04-30T06:25:47.935ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b8e0ce1-74c4-4a83-85be-fbfaf4085116", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66352025-04-30T06:25:47.935ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
66362025-04-30T06:25:47.936ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdbad5b5-e727-48ea-8166-3f19404efe74", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66372025-04-30T06:25:47.937ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
66382025-04-30T06:25:47.940ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3145eb25-c361-4b31-b23a-cd6e748ff373", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66392025-04-30T06:25:47.941ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
66402025-04-30T06:25:47.942ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46faec4f-a89d-42fa-a958-7fdc05698a61", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66412025-04-30T06:25:47.942ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
66422025-04-30T06:25:47.943ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac2bda5e-f6a0-4e79-a870-47085cef0627", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66432025-04-30T06:25:47.944ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
66442025-04-30T06:25:47.945ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c73c48-2fde-416a-a1b0-f4bed027e65f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66452025-04-30T06:25:47.945ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
66462025-04-30T06:25:47.948ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bb01449-2485-4efb-b357-7773e40dca02", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66472025-04-30T06:25:47.949ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
66482025-04-30T06:25:47.961ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66492025-04-30T06:25:47.961ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66502025-04-30T06:25:47.961ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66512025-04-30T06:25:47.961ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
66522025-04-30T06:25:47.962ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66532025-04-30T06:25:47.965ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "851e2a8d-b7e7-4e82-9fc1-759f586d5342", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66542025-04-30T06:25:47.965ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
66552025-04-30T06:25:47.966ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66562025-04-30T06:25:47.972ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66572025-04-30T06:25:47.973ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66582025-04-30T06:25:47.973ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
66592025-04-30T06:25:47.973ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66602025-04-30T06:25:47.976ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bce6a2e-72a4-4b90-92e5-add1f30e7312", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66612025-04-30T06:25:47.976ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
66622025-04-30T06:25:47.976ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66632025-04-30T06:25:47.976ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
66642025-04-30T06:25:47.977ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66652025-04-30T06:25:47.983ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f7381d-0eaa-454e-8c5d-712a2424765d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66662025-04-30T06:25:47.983ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
66672025-04-30T06:25:47.985ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aab0aabc-5523-4f73-a4bf-7af1334842f5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66682025-04-30T06:25:47.986ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
66692025-04-30T06:25:47.988ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1661e89-1e94-4bac-bf87-1c07fac1c038", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66702025-04-30T06:25:47.988ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
66712025-04-30T06:25:47.990ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8141360b-4e23-4fa9-8aeb-caf0f266be4e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66722025-04-30T06:25:47.990ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
66732025-04-30T06:25:47.992ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0760e9b-ddbc-4cb3-950a-ad85d1542632", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66742025-04-30T06:25:47.992ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
66752025-04-30T06:25:47.993ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70d9e2bb-b055-4abd-81ab-2f79345e063b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66762025-04-30T06:25:47.994ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
66772025-04-30T06:25:47.996ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2efddbe7-05a2-4b67-883f-811a1a86eedc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66782025-04-30T06:25:47.996ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
66792025-04-30T06:25:47.997ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "562389c0-61a3-4f9d-af18-5f31895fba42", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66802025-04-30T06:25:47.998ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
66812025-04-30T06:25:48.001ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "587e0ea0-c4e8-424d-88c2-9f0308b5e516", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66822025-04-30T06:25:48.001ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
66832025-04-30T06:25:48.004ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59cd1ea4-3e0e-48ed-bdab-b36499984a54", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66842025-04-30T06:25:48.005ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
66852025-04-30T06:25:48.008ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
66862025-04-30T06:25:48.009ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c4efc42-978e-4361-8092-c42d15dbd4ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66872025-04-30T06:25:48.009ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
66882025-04-30T06:25:48.012ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cb8bf15-b8eb-43d8-bed3-2b312e6c539f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66892025-04-30T06:25:48.012ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
66902025-04-30T06:25:48.014ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6318216d-b5ee-494a-a8fe-2e220a5a0d26", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66912025-04-30T06:25:48.014ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
66922025-04-30T06:25:48.017ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ebbb1a1-137b-4b98-9d87-0f0c3e8e6249", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66932025-04-30T06:25:48.017ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
66942025-04-30T06:25:48.018ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34d4ca4f-ffaf-44d7-9f5b-a06a290f0373", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66952025-04-30T06:25:48.018ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
66962025-04-30T06:25:48.019ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e904b03-fa7a-4bcf-9e5c-7f057b4644cb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66972025-04-30T06:25:48.020ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
66982025-04-30T06:25:48.021ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eddfecf-8445-4c25-b5d4-9b303443eca9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
66992025-04-30T06:25:48.021ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
67002025-04-30T06:25:48.022ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20235a34-15db-44c3-9997-188fe77f7fbe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67012025-04-30T06:25:48.022ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
67022025-04-30T06:25:48.023ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
67032025-04-30T06:25:48.024ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "39dded8f-7f80-4957-bd0d-ad9d0e537690", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67042025-04-30T06:25:48.025ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
67052025-04-30T06:25:48.026ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0833528b-e117-4c96-9cab-298bcb04c53c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67062025-04-30T06:25:48.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
67072025-04-30T06:25:48.028ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acfcd118-070a-43f4-8ebe-8a033c691e09", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67082025-04-30T06:25:48.028ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
67092025-04-30T06:25:48.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6000f6c6-2414-4602-82f8-323b3e53a49f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67102025-04-30T06:25:48.030ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
67112025-04-30T06:25:48.031ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de793f90-2c16-4c0d-8ad1-032a88ae650e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67122025-04-30T06:25:48.032ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67132025-04-30T06:25:48.032ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67142025-04-30T06:25:48.032ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67152025-04-30T06:25:48.032ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67162025-04-30T06:25:48.033ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67172025-04-30T06:25:48.035ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67182025-04-30T06:25:48.039ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67192025-04-30T06:25:48.039ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67202025-04-30T06:25:48.039ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67212025-04-30T06:25:48.039ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67222025-04-30T06:25:48.039ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67232025-04-30T06:25:48.040ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67242025-04-30T06:25:48.042ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67252025-04-30T06:25:48.055ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67262025-04-30T06:25:48.055ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67272025-04-30T06:25:48.055ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67282025-04-30T06:25:48.055ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67292025-04-30T06:25:48.134ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
67302025-04-30T06:25:48.134ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
67312025-04-30T06:25:48.135ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67322025-04-30T06:25:48.135ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67332025-04-30T06:25:48.135ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67342025-04-30T06:25:48.135ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
67352025-04-30T06:25:48.136ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67362025-04-30T06:25:48.137ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:37471 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
67372025-04-30T06:25:48.139ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:37471 req_id = 75871e89-ac35-48e6-9cc5-69868e66aaec sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
67382025-04-30T06:25:48.139ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 385 local_addr = [::1]:42668 method = GET remote_addr = [::1]:37471 req_id = 75871e89-ac35-48e6-9cc5-69868e66aaec response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
67392025-04-30T06:25:48.140ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "75871e89-ac35-48e6-9cc5-69868e66aaec", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67402025-04-30T06:25:48.196ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
67412025-04-30T06:25:48.196ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
67422025-04-30T06:25:48.197ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:37008 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
67432025-04-30T06:25:48.199ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67442025-04-30T06:25:48.205ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67452025-04-30T06:25:48.205ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67462025-04-30T06:25:48.205ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
67472025-04-30T06:25:48.205ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67482025-04-30T06:25:48.206ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:37008 req_id = 3fbe3162-cd79-4f98-80d1-9bdb58db27fe sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
67492025-04-30T06:25:48.206ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 470 local_addr = [::1]:47918 method = GET remote_addr = [::1]:37008 req_id = 3fbe3162-cd79-4f98-80d1-9bdb58db27fe response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
67502025-04-30T06:25:48.207ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fbe3162-cd79-4f98-80d1-9bdb58db27fe", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67512025-04-30T06:25:48.254ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
67522025-04-30T06:25:48.254ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
67532025-04-30T06:25:48.257ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:38512 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
67542025-04-30T06:25:48.258ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67552025-04-30T06:25:48.258ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
67562025-04-30T06:25:48.258ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67572025-04-30T06:25:48.259ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:38512 req_id = 584fed95-31b0-4d70-ae97-88d8aa6e59b5 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
67582025-04-30T06:25:48.267ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 8396 local_addr = [::1]:35744 method = GET remote_addr = [::1]:38512 req_id = 584fed95-31b0-4d70-ae97-88d8aa6e59b5 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
67592025-04-30T06:25:48.269ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "584fed95-31b0-4d70-ae97-88d8aa6e59b5", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67602025-04-30T06:25:48.342ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
67612025-04-30T06:25:48.342ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
67622025-04-30T06:25:48.344ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:61497 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
67632025-04-30T06:25:48.354ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:61497 req_id = a6d9b411-b7cb-4eea-a4db-33641b5e6e47 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67642025-04-30T06:25:48.356ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1292 local_addr = [::1]:35343 method = GET remote_addr = [::1]:61497 req_id = a6d9b411-b7cb-4eea-a4db-33641b5e6e47 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
67652025-04-30T06:25:48.364ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6d9b411-b7cb-4eea-a4db-33641b5e6e47", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:47 GMT"} })
67662025-04-30T06:25:48.368ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67672025-04-30T06:25:48.369ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
67682025-04-30T06:25:48.369ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
67692025-04-30T06:25:48.374ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
67702025-04-30T06:25:48.374ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67712025-04-30T06:25:48.374ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67722025-04-30T06:25:48.374ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67732025-04-30T06:25:48.376ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67742025-04-30T06:25:48.386ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67752025-04-30T06:25:48.395ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
67762025-04-30T06:25:48.396ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67772025-04-30T06:25:48.396ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67782025-04-30T06:25:48.396ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67792025-04-30T06:25:48.408ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67802025-04-30T06:25:48.411ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67812025-04-30T06:25:48.420ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67822025-04-30T06:25:48.420ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67832025-04-30T06:25:48.420ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67842025-04-30T06:25:48.420ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
67852025-04-30T06:25:48.446ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
67862025-04-30T06:25:48.455ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
67872025-04-30T06:25:48.456ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
67882025-04-30T06:25:48.457ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 750.719349ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
67892025-04-30T06:25:48.457ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
67902025-04-30T06:25:48.457ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
67912025-04-30T06:25:48.457ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67922025-04-30T06:25:48.457ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67932025-04-30T06:25:48.457ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67942025-04-30T06:25:48.459ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67952025-04-30T06:25:48.470ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67962025-04-30T06:25:48.477ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
67972025-04-30T06:25:48.478ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67982025-04-30T06:25:48.478ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67992025-04-30T06:25:48.481ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68002025-04-30T06:25:48.494ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
68012025-04-30T06:25:48.494ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
68022025-04-30T06:25:48.498ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 35 11 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 5............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58827 req_id = 0b633f39-501a-41ce-a35a-e9f92956b9a7
68032025-04-30T06:25:48.498ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13585,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58827 req_id = 0b633f39-501a-41ce-a35a-e9f92956b9a7
68042025-04-30T06:25:48.499ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
68052025-04-30T06:25:48.499ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
68062025-04-30T06:25:48.500ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68072025-04-30T06:25:48.500ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58827 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0b633f39-501a-41ce-a35a-e9f92956b9a7
68082025-04-30T06:25:48.501ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68092025-04-30T06:25:48.501ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68102025-04-30T06:25:48.501ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68112025-04-30T06:25:48.501ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
68122025-04-30T06:25:48.502ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68132025-04-30T06:25:48.503ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3213, tv_nsec: 302271081 } })
68142025-04-30T06:25:48.503ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
68152025-04-30T06:25:48.503ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
68162025-04-30T06:25:48.505ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5c ed 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e \\...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64661 req_id = b039f238-85b3-43c5-aec5-665203d6ea4f
68172025-04-30T06:25:48.505ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23789,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64661 req_id = b039f238-85b3-43c5-aec5-665203d6ea4f
68182025-04-30T06:25:48.505ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
68192025-04-30T06:25:48.505ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64661 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b039f238-85b3-43c5-aec5-665203d6ea4f
68202025-04-30T06:25:48.505ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 84 a9 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54877 req_id = 8e5e6226-3b27-4a9f-906d-c126c70ff502
68212025-04-30T06:25:48.506ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33961,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54877 req_id = 8e5e6226-3b27-4a9f-906d-c126c70ff502
68222025-04-30T06:25:48.506ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68232025-04-30T06:25:48.506ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54877 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8e5e6226-3b27-4a9f-906d-c126c70ff502
68242025-04-30T06:25:48.506ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68252025-04-30T06:25:48.512ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68262025-04-30T06:25:48.513ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68272025-04-30T06:25:48.513ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
68282025-04-30T06:25:48.513ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68292025-04-30T06:25:48.642ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68302025-04-30T06:25:48.644ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
68312025-04-30T06:25:48.644ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
68322025-04-30T06:25:48.645ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
68332025-04-30T06:25:48.645ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
68342025-04-30T06:25:48.647ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3d dd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c =............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64623 req_id = 91aac27f-c4be-4ac5-87d5-f095a1329244
68352025-04-30T06:25:48.647ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15837,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64623 req_id = 91aac27f-c4be-4ac5-87d5-f095a1329244
68362025-04-30T06:25:48.647ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
68372025-04-30T06:25:48.647ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15837, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64623 req_id = 91aac27f-c4be-4ac5-87d5-f095a1329244
68382025-04-30T06:25:48.648ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
68392025-04-30T06:25:48.648ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
68402025-04-30T06:25:48.648ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
68412025-04-30T06:25:48.648ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68422025-04-30T06:25:48.648ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
68432025-04-30T06:25:48.650ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68442025-04-30T06:25:48.655ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
68452025-04-30T06:25:48.655ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
68462025-04-30T06:25:48.657ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f390b1-4cd4-454b-9493-c727b49ea76e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68472025-04-30T06:25:48.657ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
68482025-04-30T06:25:48.667ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8e7bbcd-c61a-4297-9e21-9a53d71f5ff2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68492025-04-30T06:25:48.667ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
68502025-04-30T06:25:48.667ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
68512025-04-30T06:25:48.667ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
68522025-04-30T06:25:48.672ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dbccc97-8fcf-4b68-b75b-9951da07cfd1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68532025-04-30T06:25:48.672ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
68542025-04-30T06:25:48.676ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49d32f94-8fb3-422a-a303-8c0223e590db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68552025-04-30T06:25:48.678ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
68562025-04-30T06:25:48.682ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7db8e903-bb6c-4546-9211-727a604ed40f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68572025-04-30T06:25:48.682ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
68582025-04-30T06:25:48.688ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d384d36-3a4b-4489-83fe-02d4d7639ffc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68592025-04-30T06:25:48.688ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
68602025-04-30T06:25:48.692ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
68612025-04-30T06:25:48.697ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fd54bff-dfa0-454e-9231-ae8e36bfb2cf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68622025-04-30T06:25:48.697ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
68632025-04-30T06:25:48.701ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10c93505-78bd-4ee8-8cda-b29ad520d910", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68642025-04-30T06:25:48.701ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
68652025-04-30T06:25:48.704ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b466829-2dc7-4996-8208-c9ecd05a5aea", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68662025-04-30T06:25:48.704ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
68672025-04-30T06:25:48.706ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eadde737-9419-489b-a07a-cc588b09f877", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68682025-04-30T06:25:48.707ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
68692025-04-30T06:25:48.711ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1fc619a-bc72-4135-8120-3ed1c5c27f43", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68702025-04-30T06:25:48.711ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
68712025-04-30T06:25:48.715ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed804567-30b0-4a67-94fa-9452949cbc5b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68722025-04-30T06:25:48.716ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
68732025-04-30T06:25:48.717ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "71c9d83b-60d5-4e54-ab28-e633afe7ad86", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68742025-04-30T06:25:48.717ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
68752025-04-30T06:25:48.720ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efad3af1-800f-49b0-b805-58b71e58cbd3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68762025-04-30T06:25:48.720ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
68772025-04-30T06:25:48.734ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d271b58d-7703-4fa3-8e4e-1f009735c8b0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68782025-04-30T06:25:48.734ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
68792025-04-30T06:25:48.743ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "be57ae17-b884-44b7-8bd7-0e0e17ed04e1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68802025-04-30T06:25:48.743ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
68812025-04-30T06:25:48.744ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68822025-04-30T06:25:48.744ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68832025-04-30T06:25:48.744ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68842025-04-30T06:25:48.744ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
68852025-04-30T06:25:48.746ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68862025-04-30T06:25:48.747ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
68872025-04-30T06:25:48.749ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68882025-04-30T06:25:48.755ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68892025-04-30T06:25:48.755ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68902025-04-30T06:25:48.756ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
68912025-04-30T06:25:48.756ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68922025-04-30T06:25:48.758ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "173f688c-14d3-4766-88ff-5a21ccb1f48e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68932025-04-30T06:25:48.758ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
68942025-04-30T06:25:48.759ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68952025-04-30T06:25:48.759ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
68962025-04-30T06:25:48.759ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68972025-04-30T06:25:48.783ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e621d681-5f54-4a47-a103-6ac0186569f8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
68982025-04-30T06:25:48.784ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
68992025-04-30T06:25:48.788ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "faaba0f9-8d0a-48f9-a2df-681144026ca9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69002025-04-30T06:25:48.788ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
69012025-04-30T06:25:48.791ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "572ec784-3912-41af-ac5a-15e013fc4581", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69022025-04-30T06:25:48.791ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
69032025-04-30T06:25:48.794ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9334caee-39dd-4be4-9084-d84c579337ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69042025-04-30T06:25:48.794ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
69052025-04-30T06:25:48.795ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "309ae44f-838b-4410-931e-cbbc6c640f90", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69062025-04-30T06:25:48.796ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
69072025-04-30T06:25:48.798ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a91a60a-cdc8-4bb3-b06e-90de6e4e190a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69082025-04-30T06:25:48.798ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
69092025-04-30T06:25:48.800ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52adaf10-edbd-4bce-bf25-74222364a102", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69102025-04-30T06:25:48.800ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
69112025-04-30T06:25:48.803ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3347e5e5-6122-4eaf-9697-8a9678ab35ab", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69122025-04-30T06:25:48.803ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
69132025-04-30T06:25:48.808ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5da1472-b1c7-4548-9bb4-4c2ab5926a8d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69142025-04-30T06:25:48.808ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
69152025-04-30T06:25:48.808ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
69162025-04-30T06:25:48.811ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a179217-328b-42a4-9b36-0ee21d31aeac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69172025-04-30T06:25:48.811ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
69182025-04-30T06:25:48.814ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9f4e1d6-901f-44f3-8b38-99ebf3477394", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69192025-04-30T06:25:48.814ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
69202025-04-30T06:25:48.816ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c73efd03-1429-4277-9081-d17c0cd77e4e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69212025-04-30T06:25:48.816ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
69222025-04-30T06:25:48.820ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "def26e8a-57c4-4f43-83c1-7ce9e7f280e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69232025-04-30T06:25:48.820ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
69242025-04-30T06:25:48.821ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "34ec3136-0b44-4dfd-b0eb-d36752f73f3c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69252025-04-30T06:25:48.821ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
69262025-04-30T06:25:48.823ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb6c2b82-5128-49fa-b383-96a840eaf4e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69272025-04-30T06:25:48.823ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
69282025-04-30T06:25:48.824ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9919b9d8-e604-4b06-9c2c-5802fe625c3d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69292025-04-30T06:25:48.824ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
69302025-04-30T06:25:48.825ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4389d54b-243e-49b5-b48b-27f540f3a6b3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69312025-04-30T06:25:48.826ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
69322025-04-30T06:25:48.827ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8909eab3-ced7-4dd2-88d3-369ff88158d3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69332025-04-30T06:25:48.827ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
69342025-04-30T06:25:48.831ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "756d7b87-e884-4d09-878b-adc158c30913", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69352025-04-30T06:25:48.831ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
69362025-04-30T06:25:48.836ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c799ccb5-ffb5-4a96-bd4c-95ca5bc9357e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69372025-04-30T06:25:48.836ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
69382025-04-30T06:25:48.841ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fcd092f-8745-42bc-9f54-3c5f9118409a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69392025-04-30T06:25:48.841ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
69402025-04-30T06:25:48.844ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09020482-fb99-4618-a1e9-35b9d68e807a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69412025-04-30T06:25:48.845ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
69422025-04-30T06:25:48.847ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99a8ff99-ce43-4b82-b900-665d3b5428b7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69432025-04-30T06:25:48.848ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
69442025-04-30T06:25:48.851ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d9f8a2b-7ab5-4bb2-80da-f480ed8ce8dd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69452025-04-30T06:25:48.851ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
69462025-04-30T06:25:48.852ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b645ce10-115b-4722-b5c5-70d11c23a71d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69472025-04-30T06:25:48.853ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
69482025-04-30T06:25:48.856ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f1f5438-d6cf-40ca-abab-d696a6511393", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69492025-04-30T06:25:48.856ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
69502025-04-30T06:25:48.858ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d9fa2a8-959f-4573-b200-0ab0920e5135", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69512025-04-30T06:25:48.863ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
69522025-04-30T06:25:48.878ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69532025-04-30T06:25:48.878ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69542025-04-30T06:25:48.878ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69552025-04-30T06:25:48.878ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
69562025-04-30T06:25:48.881ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69572025-04-30T06:25:48.886ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69582025-04-30T06:25:48.893ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69592025-04-30T06:25:48.893ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69602025-04-30T06:25:48.893ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
69612025-04-30T06:25:48.894ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69622025-04-30T06:25:48.894ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fc28655-5b7c-4dec-9572-d8327d7d5467", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69632025-04-30T06:25:48.894ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
69642025-04-30T06:25:48.895ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
69652025-04-30T06:25:48.901ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69662025-04-30T06:25:48.901ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
69672025-04-30T06:25:48.903ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69682025-04-30T06:25:48.906ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "43cedf64-7e16-4bb5-af84-251368464b10", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69692025-04-30T06:25:48.907ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
69702025-04-30T06:25:48.910ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "feae64da-5dc3-4a07-b6e2-e5d2c86460e7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69712025-04-30T06:25:48.910ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
69722025-04-30T06:25:48.912ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9eabc52-7687-404e-885b-d80f0f0b37a8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69732025-04-30T06:25:48.912ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
69742025-04-30T06:25:48.920ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9853712d-f572-4d62-b01d-e1c54616a941", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69752025-04-30T06:25:48.920ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
69762025-04-30T06:25:48.923ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3393c36-48e0-465b-ae34-1d209c7c818e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
69772025-04-30T06:25:48.923ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69782025-04-30T06:25:48.923ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69792025-04-30T06:25:48.923ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69802025-04-30T06:25:48.923ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69812025-04-30T06:25:48.924ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69822025-04-30T06:25:48.931ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69832025-04-30T06:25:48.936ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69842025-04-30T06:25:48.936ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69852025-04-30T06:25:48.937ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69862025-04-30T06:25:48.937ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69872025-04-30T06:25:48.937ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69882025-04-30T06:25:48.938ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69892025-04-30T06:25:48.941ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69902025-04-30T06:25:48.946ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69912025-04-30T06:25:48.946ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69922025-04-30T06:25:48.946ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69932025-04-30T06:25:48.947ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69942025-04-30T06:25:49.023ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
69952025-04-30T06:25:49.023ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
69962025-04-30T06:25:49.024ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:56613 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
69972025-04-30T06:25:49.025ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:56613 req_id = d1faeddb-5ae8-4ca8-8517-92db6d8f359c sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
69982025-04-30T06:25:49.025ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 846 local_addr = [::1]:42668 method = GET remote_addr = [::1]:56613 req_id = d1faeddb-5ae8-4ca8-8517-92db6d8f359c response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
69992025-04-30T06:25:49.026ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1faeddb-5ae8-4ca8-8517-92db6d8f359c", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
70002025-04-30T06:25:49.084ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
70012025-04-30T06:25:49.084ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
70022025-04-30T06:25:49.086ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:37320 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
70032025-04-30T06:25:49.086ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:37320 req_id = e8787177-1209-46e4-9d4b-bc02269d244a sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
70042025-04-30T06:25:49.087ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 385 local_addr = [::1]:47918 method = GET remote_addr = [::1]:37320 req_id = e8787177-1209-46e4-9d4b-bc02269d244a response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
70052025-04-30T06:25:49.087ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8787177-1209-46e4-9d4b-bc02269d244a", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
70062025-04-30T06:25:49.158ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
70072025-04-30T06:25:49.158ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
70082025-04-30T06:25:49.159ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:36617 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
70092025-04-30T06:25:49.160ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:36617 req_id = 36472501-fe7c-4c60-bce8-712172c39112 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
70102025-04-30T06:25:49.160ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 391 local_addr = [::1]:35744 method = GET remote_addr = [::1]:36617 req_id = 36472501-fe7c-4c60-bce8-712172c39112 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
70112025-04-30T06:25:49.161ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "36472501-fe7c-4c60-bce8-712172c39112", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
70122025-04-30T06:25:49.207ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
70132025-04-30T06:25:49.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
70142025-04-30T06:25:49.213ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:54896 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70152025-04-30T06:25:49.215ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
70162025-04-30T06:25:49.215ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:54896 req_id = fbe1dced-81b8-49b9-836e-93f7e251a853 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70172025-04-30T06:25:49.229ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 13170 local_addr = [::1]:35343 method = GET remote_addr = [::1]:54896 req_id = fbe1dced-81b8-49b9-836e-93f7e251a853 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70182025-04-30T06:25:49.232ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbe1dced-81b8-49b9-836e-93f7e251a853", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
70192025-04-30T06:25:49.240ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70202025-04-30T06:25:49.240ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70212025-04-30T06:25:49.240ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
70222025-04-30T06:25:49.258ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70232025-04-30T06:25:49.258ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70242025-04-30T06:25:49.258ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70252025-04-30T06:25:49.258ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70262025-04-30T06:25:49.259ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70272025-04-30T06:25:49.270ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70282025-04-30T06:25:49.270ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70292025-04-30T06:25:49.270ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70302025-04-30T06:25:49.270ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
70312025-04-30T06:25:49.271ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70322025-04-30T06:25:49.272ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70332025-04-30T06:25:49.277ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70342025-04-30T06:25:49.277ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70352025-04-30T06:25:49.277ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70362025-04-30T06:25:49.277ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70372025-04-30T06:25:49.285ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70382025-04-30T06:25:49.290ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70392025-04-30T06:25:49.290ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70402025-04-30T06:25:49.290ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
70412025-04-30T06:25:49.291ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70422025-04-30T06:25:49.298ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70432025-04-30T06:25:49.298ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
70442025-04-30T06:25:49.298ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70452025-04-30T06:25:49.299ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
70462025-04-30T06:25:49.299ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70472025-04-30T06:25:49.299ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 1f34e4de-6724-478f-8ce1-070052a5bf04 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70482025-04-30T06:25:49.303ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 1f34e4de-6724-478f-8ce1-070052a5bf04 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70492025-04-30T06:25:49.303ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 1f34e4de-6724-478f-8ce1-070052a5bf04 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70502025-04-30T06:25:49.303ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 1f34e4de-6724-478f-8ce1-070052a5bf04 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70512025-04-30T06:25:49.307ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 7026 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 1f34e4de-6724-478f-8ce1-070052a5bf04 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70522025-04-30T06:25:49.308ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f34e4de-6724-478f-8ce1-070052a5bf04", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
70532025-04-30T06:25:49.308ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
70542025-04-30T06:25:49.309ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b34f7090-0171-4b3c-8fe6-6997834f4771 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
70552025-04-30T06:25:49.309ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b34f7090-0171-4b3c-8fe6-6997834f4771 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
70562025-04-30T06:25:49.309ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b34f7090-0171-4b3c-8fe6-6997834f4771 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
70572025-04-30T06:25:49.310ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b34f7090-0171-4b3c-8fe6-6997834f4771 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
70582025-04-30T06:25:49.321ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 11903 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b34f7090-0171-4b3c-8fe6-6997834f4771 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
70592025-04-30T06:25:49.322ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "b34f7090-0171-4b3c-8fe6-6997834f4771", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:48 GMT"} })
70602025-04-30T06:25:49.322ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70612025-04-30T06:25:49.322ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
70622025-04-30T06:25:49.322ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
70632025-04-30T06:25:49.338ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
70642025-04-30T06:25:49.338ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
70652025-04-30T06:25:49.339ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 881.656413ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
70662025-04-30T06:25:49.339ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
70672025-04-30T06:25:49.340ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70682025-04-30T06:25:49.340ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70692025-04-30T06:25:49.340ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70702025-04-30T06:25:49.340ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70712025-04-30T06:25:49.342ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70722025-04-30T06:25:49.346ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70732025-04-30T06:25:49.353ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70742025-04-30T06:25:49.353ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70752025-04-30T06:25:49.353ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70762025-04-30T06:25:49.354ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70772025-04-30T06:25:49.382ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
70782025-04-30T06:25:49.382ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
70792025-04-30T06:25:49.384ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 58 b5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 X............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55630 req_id = 6e8536aa-8122-4399-bc88-aa7410c80504
70802025-04-30T06:25:49.384ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22709,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55630 req_id = 6e8536aa-8122-4399-bc88-aa7410c80504
70812025-04-30T06:25:49.384ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
70822025-04-30T06:25:49.385ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
70832025-04-30T06:25:49.385ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70842025-04-30T06:25:49.385ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55630 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6e8536aa-8122-4399-bc88-aa7410c80504
70852025-04-30T06:25:49.385ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
70862025-04-30T06:25:49.386ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3214, tv_nsec: 184941805 } })
70872025-04-30T06:25:49.386ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
70882025-04-30T06:25:49.386ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
70892025-04-30T06:25:49.387ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 79 de 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e y...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49334 req_id = f514c47a-7b9f-428b-8fbf-3ddb096226ea
70902025-04-30T06:25:49.387ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31198,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49334 req_id = f514c47a-7b9f-428b-8fbf-3ddb096226ea
70912025-04-30T06:25:49.387ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
70922025-04-30T06:25:49.387ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49334 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f514c47a-7b9f-428b-8fbf-3ddb096226ea
70932025-04-30T06:25:49.387ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bf 10 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60954 req_id = bda16fb9-b688-4fdb-a802-9c7c7a349244
70942025-04-30T06:25:49.387ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48912,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60954 req_id = bda16fb9-b688-4fdb-a802-9c7c7a349244
70952025-04-30T06:25:49.387ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70962025-04-30T06:25:49.387ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60954 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = bda16fb9-b688-4fdb-a802-9c7c7a349244
70972025-04-30T06:25:49.537ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
70982025-04-30T06:25:49.538ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70992025-04-30T06:25:49.538ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71002025-04-30T06:25:49.538ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71012025-04-30T06:25:49.538ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
71022025-04-30T06:25:49.539ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71032025-04-30T06:25:49.539ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71042025-04-30T06:25:49.539ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
71052025-04-30T06:25:49.539ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71062025-04-30T06:25:49.539ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
71072025-04-30T06:25:49.540ZINFOtest_replacement_sanity (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:50502 remote_addr = 127.0.0.1:65326
71082025-04-30T06:25:49.541ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6f 37 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c o7..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58396 req_id = b1c4d55f-5f26-433d-9ab0-f012f4c672df
71092025-04-30T06:25:49.541ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28471,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58396 req_id = b1c4d55f-5f26-433d-9ab0-f012f4c672df
71102025-04-30T06:25:49.541ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
71112025-04-30T06:25:49.541ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28471, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58396 req_id = b1c4d55f-5f26-433d-9ab0-f012f4c672df
71122025-04-30T06:25:49.544ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
71132025-04-30T06:25:49.544ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
71142025-04-30T06:25:49.544ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
71152025-04-30T06:25:49.546ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = 127.0.0.1:50502 method = GET remote_addr = 127.0.0.1:65326 req_id = 270435d1-e0a2-4b80-bc9d-5610ddb4b5e9 uri = /c31da0ae-930d-4dd9-a897-159d5a6cd420
71162025-04-30T06:25:49.559ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71172025-04-30T06:25:49.559ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71182025-04-30T06:25:49.559ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71192025-04-30T06:25:49.560ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
71202025-04-30T06:25:49.560ZINFOtest_replacement_sanity (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 13498 local_addr = 127.0.0.1:50502 method = GET remote_addr = 127.0.0.1:65326 req_id = 270435d1-e0a2-4b80-bc9d-5610ddb4b5e9 response_code = 200 uri = /c31da0ae-930d-4dd9-a897-159d5a6cd420
71212025-04-30T06:25:49.560ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71222025-04-30T06:25:49.578ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71232025-04-30T06:25:49.578ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71242025-04-30T06:25:49.578ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
71252025-04-30T06:25:49.600ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71262025-04-30T06:25:49.607ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
71272025-04-30T06:25:49.623ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
71282025-04-30T06:25:49.623ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
71292025-04-30T06:25:49.623ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
71302025-04-30T06:25:49.624ZDEBGtest_replacement_sanity (results-sink): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71312025-04-30T06:25:49.624ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 18 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71322025-04-30T06:25:49.624ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 18 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71332025-04-30T06:25:49.628ZDEBGtest_replacement_sanity (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71342025-04-30T06:25:49.629ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
71352025-04-30T06:25:49.641ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71362025-04-30T06:25:49.641ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
71372025-04-30T06:25:49.642ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71382025-04-30T06:25:49.643ZTRACtest_replacement_sanity (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71392025-04-30T06:25:49.643ZDEBGtest_replacement_sanity (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:25:49.628872233Z } timeseries_name = http_service:request_latency_histogram
71402025-04-30T06:25:49.647ZDEBGtest_replacement_sanity (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71412025-04-30T06:25:49.648ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
71422025-04-30T06:25:49.660ZTRACtest_replacement_sanity (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71432025-04-30T06:25:49.660ZDEBGtest_replacement_sanity (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:25:49.647887189Z } timeseries_name = collection_target:cpus_provisioned
71442025-04-30T06:25:49.661ZDEBGtest_replacement_sanity (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71452025-04-30T06:25:49.661ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
71462025-04-30T06:25:49.666ZTRACtest_replacement_sanity (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71472025-04-30T06:25:49.666ZDEBGtest_replacement_sanity (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:25:49.661792584Z } timeseries_name = collection_target:ram_provisioned
71482025-04-30T06:25:49.666ZDEBGtest_replacement_sanity (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71492025-04-30T06:25:49.667ZTRACtest_replacement_sanity (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
71502025-04-30T06:25:49.675ZTRACtest_replacement_sanity (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71512025-04-30T06:25:49.675ZDEBGtest_replacement_sanity (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:25:49.666919840Z } timeseries_name = collection_target:virtual_disk_space_provisioned
71522025-04-30T06:25:49.683ZDEBGtest_replacement_sanity (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
71532025-04-30T06:25:49.683ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
71542025-04-30T06:25:49.683ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "80235308-eef8-4071-ba85-9df60449b952", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71552025-04-30T06:25:49.684ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
71562025-04-30T06:25:49.694ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
71572025-04-30T06:25:49.698ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6bb0344-4155-4d6a-b28f-ddd17c48c83d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71582025-04-30T06:25:49.698ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
71592025-04-30T06:25:49.707ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 12 table_name = fields_string
71602025-04-30T06:25:49.707ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
71612025-04-30T06:25:49.713ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 5 table_name = fields_u16
71622025-04-30T06:25:49.713ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 17 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
71632025-04-30T06:25:49.717ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffecadea-16c7-4196-8a92-c47e55f09b8c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71642025-04-30T06:25:49.717ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
71652025-04-30T06:25:49.718ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 17 table_name = fields_uuid
71662025-04-30T06:25:49.718ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
71672025-04-30T06:25:49.725ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 5 table_name = measurements_histogramu64
71682025-04-30T06:25:49.725ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 13 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
71692025-04-30T06:25:49.725ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
71702025-04-30T06:25:49.726ZTRACtest_replacement_sanity (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71712025-04-30T06:25:49.726ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
71722025-04-30T06:25:49.730ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1920d890-5177-4efd-9a83-56059c76e8d4", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71732025-04-30T06:25:49.731ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
71742025-04-30T06:25:49.732ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 13 table_name = measurements_i64
71752025-04-30T06:25:49.732ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71762025-04-30T06:25:49.733ZTRACtest_replacement_sanity (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71772025-04-30T06:25:49.737ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aaffa644-dbd8-45ea-a956-ce88ca708a50", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71782025-04-30T06:25:49.737ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
71792025-04-30T06:25:49.743ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd98873c-a7db-4eb9-afd2-d0b01fe554b7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71802025-04-30T06:25:49.744ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
71812025-04-30T06:25:49.746ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf594388-0c13-4d2f-b236-4a38aa012aef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71822025-04-30T06:25:49.747ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
71832025-04-30T06:25:49.751ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7036e418-d5ff-4ede-8e59-29f36d2ed914", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71842025-04-30T06:25:49.751ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
71852025-04-30T06:25:49.753ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "392e4360-8c7e-4d23-9d78-a3b8c3f8d7a7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71862025-04-30T06:25:49.753ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
71872025-04-30T06:25:49.756ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1364d83d-ab9f-432b-abe9-434ff5346de4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71882025-04-30T06:25:49.756ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
71892025-04-30T06:25:49.757ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd656557-efb3-499e-812f-84db11dd4fae", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71902025-04-30T06:25:49.758ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
71912025-04-30T06:25:49.761ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e61e72a0-86b9-4b70-ab9e-ce266c2d46db", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71922025-04-30T06:25:49.763ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
71932025-04-30T06:25:49.764ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0c104ac-386b-426c-baf6-d3ada7a6549a", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71942025-04-30T06:25:49.765ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
71952025-04-30T06:25:49.769ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "529b9737-4814-4713-814d-7f3c71902545", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71962025-04-30T06:25:49.769ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
71972025-04-30T06:25:49.771ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af1a35c2-849b-42b0-91ba-e716f9b0fa26", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
71982025-04-30T06:25:49.771ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
71992025-04-30T06:25:49.776ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b86c8bb-f841-4227-a91f-2e6b8db24c56", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72002025-04-30T06:25:49.776ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
72012025-04-30T06:25:49.777ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72022025-04-30T06:25:49.777ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72032025-04-30T06:25:49.777ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72042025-04-30T06:25:49.777ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
72052025-04-30T06:25:49.780ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72062025-04-30T06:25:49.784ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72072025-04-30T06:25:49.795ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72082025-04-30T06:25:49.795ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72092025-04-30T06:25:49.795ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
72102025-04-30T06:25:49.797ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72112025-04-30T06:25:49.802ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e012ec0d-08d0-43f4-978d-757137e6ed57", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72122025-04-30T06:25:49.803ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
72132025-04-30T06:25:49.803ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72142025-04-30T06:25:49.803ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
72152025-04-30T06:25:49.804ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72162025-04-30T06:25:49.809ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cfaa7e3-117a-4d26-8c80-2c440dc65f96", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72172025-04-30T06:25:49.809ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
72182025-04-30T06:25:49.811ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5723369d-0bc8-43bd-aefb-cf1a8a0c91d0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72192025-04-30T06:25:49.812ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
72202025-04-30T06:25:49.815ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ef10964-db28-4cc9-9818-3fe4858374a6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72212025-04-30T06:25:49.815ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
72222025-04-30T06:25:49.817ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27397889-39b6-4721-8a00-875dea92db8b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72232025-04-30T06:25:49.817ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
72242025-04-30T06:25:49.818ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d5cbe3-ee6b-49fc-962f-e1b7e5d79c82", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72252025-04-30T06:25:49.818ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
72262025-04-30T06:25:49.819ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9822d807-d014-41f6-9aa4-76737319d91f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72272025-04-30T06:25:49.819ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
72282025-04-30T06:25:49.821ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cc0cd7b-eafa-43e4-a286-b4efde80c438", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72292025-04-30T06:25:49.822ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
72302025-04-30T06:25:49.824ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a69166cd-c55a-49b5-b191-4a376cc4da02", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72312025-04-30T06:25:49.825ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
72322025-04-30T06:25:49.828ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00bde4a7-466b-4747-b393-c53e5774ca8b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72332025-04-30T06:25:49.829ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
72342025-04-30T06:25:49.833ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b019156-69d5-4987-b190-a68351605fe7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72352025-04-30T06:25:49.834ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
72362025-04-30T06:25:49.834ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
72372025-04-30T06:25:49.837ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "824484fb-e689-4b60-acbc-f6582e510944", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72382025-04-30T06:25:49.837ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
72392025-04-30T06:25:49.840ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38174792-04f1-4daa-8fd3-811fec9a57b6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72402025-04-30T06:25:49.840ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
72412025-04-30T06:25:49.843ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "445ebb72-71d8-4356-964e-87cf61750c1b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72422025-04-30T06:25:49.843ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
72432025-04-30T06:25:49.845ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3a8dd6c-592b-41be-bb35-c8b676139d5f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72442025-04-30T06:25:49.845ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
72452025-04-30T06:25:49.846ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a7f2cfe-c35c-4438-9376-839f0175601d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72462025-04-30T06:25:49.846ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
72472025-04-30T06:25:49.847ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e184c49-4f38-40aa-9725-92d80e34ec1b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72482025-04-30T06:25:49.847ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
72492025-04-30T06:25:49.848ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e298b7b-ed51-44ed-ae51-acdef0e5f4ef", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72502025-04-30T06:25:49.849ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
72512025-04-30T06:25:49.850ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9a3ccc5-19cf-4500-8eb5-61e4d7fde2b1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72522025-04-30T06:25:49.850ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
72532025-04-30T06:25:49.852ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "10794eb1-ea51-4c93-9123-5ee165bd19fb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72542025-04-30T06:25:49.852ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
72552025-04-30T06:25:49.854ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f65f4471-9ae9-4ca2-96e2-64dc15e5c05d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72562025-04-30T06:25:49.856ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
72572025-04-30T06:25:49.856ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afae20ed-b75a-4313-bdbd-885c6b162800", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72582025-04-30T06:25:49.856ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
72592025-04-30T06:25:49.858ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f0d22b7-1592-4095-92a8-bc5ba0003e51", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72602025-04-30T06:25:49.858ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
72612025-04-30T06:25:49.860ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65f899dc-c866-4717-ab60-d522b21e9298", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72622025-04-30T06:25:49.860ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
72632025-04-30T06:25:49.862ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a130c73-3d2b-4b66-a435-c06277b6f0c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72642025-04-30T06:25:49.862ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
72652025-04-30T06:25:49.863ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "112e82c9-dfe0-40d6-b0b0-c92328a8be90", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72662025-04-30T06:25:49.863ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
72672025-04-30T06:25:49.864ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c11e7560-b2cb-4a3a-abe2-c953f7888f8f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72682025-04-30T06:25:49.865ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
72692025-04-30T06:25:49.866ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35fbeba4-34b3-4168-aa60-d345122e8cf4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72702025-04-30T06:25:49.866ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
72712025-04-30T06:25:49.867ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28cbe96c-15fe-4087-ad04-5cb40b355abe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72722025-04-30T06:25:49.867ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
72732025-04-30T06:25:49.867ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
72742025-04-30T06:25:49.869ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2216951-b3f2-4068-a948-5d03e1c69fc9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72752025-04-30T06:25:49.869ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
72762025-04-30T06:25:49.871ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2abb675-2119-423a-8a49-446a856aacbd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72772025-04-30T06:25:49.871ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
72782025-04-30T06:25:49.874ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb0cb8e1-4b05-4aef-8766-f3f84db3d40f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72792025-04-30T06:25:49.874ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
72802025-04-30T06:25:49.879ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb475a37-59ad-40f6-8273-bcfcaae46d0b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72812025-04-30T06:25:49.879ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
72822025-04-30T06:25:49.880ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5958b8b-21ba-47ef-a4ff-621e6cb9ac22", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
72832025-04-30T06:25:49.881ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72842025-04-30T06:25:49.881ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72852025-04-30T06:25:49.881ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72862025-04-30T06:25:49.881ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72872025-04-30T06:25:49.882ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72882025-04-30T06:25:49.892ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72892025-04-30T06:25:49.892ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72902025-04-30T06:25:49.892ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72912025-04-30T06:25:49.893ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
72922025-04-30T06:25:49.894ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72932025-04-30T06:25:49.897ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72942025-04-30T06:25:49.902ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72952025-04-30T06:25:49.902ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72962025-04-30T06:25:49.902ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72972025-04-30T06:25:49.902ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72982025-04-30T06:25:49.902ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72992025-04-30T06:25:49.902ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73002025-04-30T06:25:49.905ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73012025-04-30T06:25:49.915ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73022025-04-30T06:25:49.915ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73032025-04-30T06:25:49.915ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
73042025-04-30T06:25:49.915ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73052025-04-30T06:25:49.917ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73062025-04-30T06:25:49.921ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
73072025-04-30T06:25:49.921ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73082025-04-30T06:25:49.921ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73092025-04-30T06:25:49.922ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73102025-04-30T06:25:49.922ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73112025-04-30T06:25:49.922ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
73122025-04-30T06:25:49.924ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73132025-04-30T06:25:50.031ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
73142025-04-30T06:25:50.031ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
73152025-04-30T06:25:50.032ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:59520 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
73162025-04-30T06:25:50.041ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:59520 req_id = fa6b88b1-8bf2-423c-92a3-8ff1dd5fa414 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
73172025-04-30T06:25:50.043ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1867 local_addr = [::1]:42668 method = GET remote_addr = [::1]:59520 req_id = fa6b88b1-8bf2-423c-92a3-8ff1dd5fa414 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
73182025-04-30T06:25:50.044ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa6b88b1-8bf2-423c-92a3-8ff1dd5fa414", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
73192025-04-30T06:25:50.158ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
73202025-04-30T06:25:50.158ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
73212025-04-30T06:25:50.159ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:65332 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
73222025-04-30T06:25:50.160ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:65332 req_id = 913022f3-a6d3-4c19-82b7-a53bb49c66dd sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
73232025-04-30T06:25:50.162ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 430 local_addr = [::1]:47918 method = GET remote_addr = [::1]:65332 req_id = 913022f3-a6d3-4c19-82b7-a53bb49c66dd response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
73242025-04-30T06:25:50.162ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "913022f3-a6d3-4c19-82b7-a53bb49c66dd", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
73252025-04-30T06:25:50.256ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
73262025-04-30T06:25:50.256ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
73272025-04-30T06:25:50.258ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:55510 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
73282025-04-30T06:25:50.260ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:55510 req_id = 7125f5ce-c4c0-490c-b0be-b60c569154cc sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
73292025-04-30T06:25:50.271ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 11234 local_addr = [::1]:35744 method = GET remote_addr = [::1]:55510 req_id = 7125f5ce-c4c0-490c-b0be-b60c569154cc response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
73302025-04-30T06:25:50.273ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7125f5ce-c4c0-490c-b0be-b60c569154cc", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
73312025-04-30T06:25:50.531ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
73322025-04-30T06:25:50.531ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
73332025-04-30T06:25:50.533ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:47595 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
73342025-04-30T06:25:50.533ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73352025-04-30T06:25:50.538ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73362025-04-30T06:25:50.540ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:47595 req_id = 323eede7-18fb-4078-9fee-acfe730ceaa8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73372025-04-30T06:25:50.542ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1593 local_addr = [::1]:35343 method = GET remote_addr = [::1]:47595 req_id = 323eede7-18fb-4078-9fee-acfe730ceaa8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
73382025-04-30T06:25:50.549ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73392025-04-30T06:25:50.549ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73402025-04-30T06:25:50.549ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73412025-04-30T06:25:50.549ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
73422025-04-30T06:25:50.549ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "323eede7-18fb-4078-9fee-acfe730ceaa8", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:49 GMT"} })
73432025-04-30T06:25:50.565ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73442025-04-30T06:25:50.565ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
73452025-04-30T06:25:50.565ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
73462025-04-30T06:25:50.571ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73472025-04-30T06:25:50.571ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73482025-04-30T06:25:50.571ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73492025-04-30T06:25:50.571ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73502025-04-30T06:25:50.572ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73512025-04-30T06:25:50.581ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73522025-04-30T06:25:50.587ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73532025-04-30T06:25:50.587ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73542025-04-30T06:25:50.587ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73552025-04-30T06:25:50.588ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73562025-04-30T06:25:50.612ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
73572025-04-30T06:25:50.613ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73582025-04-30T06:25:50.613ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73592025-04-30T06:25:50.614ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
73602025-04-30T06:25:50.614ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
73612025-04-30T06:25:50.630ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
73622025-04-30T06:25:50.630ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
73632025-04-30T06:25:50.639ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
73642025-04-30T06:25:50.639ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
73652025-04-30T06:25:50.650ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
73662025-04-30T06:25:50.651ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
73672025-04-30T06:25:50.652ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.313235787s iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
73682025-04-30T06:25:50.653ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
73692025-04-30T06:25:50.653ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
73702025-04-30T06:25:50.653ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73712025-04-30T06:25:50.653ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73722025-04-30T06:25:50.653ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73732025-04-30T06:25:50.654ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73742025-04-30T06:25:50.658ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73752025-04-30T06:25:50.663ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
73762025-04-30T06:25:50.665ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
73772025-04-30T06:25:50.665ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
73782025-04-30T06:25:50.668ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
73792025-04-30T06:25:50.669ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73802025-04-30T06:25:50.669ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73812025-04-30T06:25:50.669ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73822025-04-30T06:25:50.669ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
73832025-04-30T06:25:50.669ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73842025-04-30T06:25:50.678ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73852025-04-30T06:25:50.694ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73862025-04-30T06:25:50.694ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73872025-04-30T06:25:50.694ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
73882025-04-30T06:25:50.695ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73892025-04-30T06:25:50.696ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
73902025-04-30T06:25:50.696ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
73912025-04-30T06:25:50.697ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 5a fd 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Z............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53554 req_id = 8f10225e-5f3e-429a-acc2-f344b9d2c76c
73922025-04-30T06:25:50.697ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23293,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53554 req_id = 8f10225e-5f3e-429a-acc2-f344b9d2c76c
73932025-04-30T06:25:50.704ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
73942025-04-30T06:25:50.704ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
73952025-04-30T06:25:50.704ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
73962025-04-30T06:25:50.704ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53554 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8f10225e-5f3e-429a-acc2-f344b9d2c76c
73972025-04-30T06:25:50.706ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3215, tv_nsec: 504993291 } })
73982025-04-30T06:25:50.706ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
73992025-04-30T06:25:50.706ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
74002025-04-30T06:25:50.706ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74012025-04-30T06:25:50.707ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
74022025-04-30T06:25:50.707ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74032025-04-30T06:25:50.708ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ad ee 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63538 req_id = 6e28efe2-49c5-4a35-bea9-e2a2ebbe8da1
74042025-04-30T06:25:50.708ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44526,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63538 req_id = 6e28efe2-49c5-4a35-bea9-e2a2ebbe8da1
74052025-04-30T06:25:50.708ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
74062025-04-30T06:25:50.708ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63538 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6e28efe2-49c5-4a35-bea9-e2a2ebbe8da1
74072025-04-30T06:25:50.709ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c3 81 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60972 req_id = c5b65cf5-b7b3-4dab-b697-44b3d507b879
74082025-04-30T06:25:50.709ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50049,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60972 req_id = c5b65cf5-b7b3-4dab-b697-44b3d507b879
74092025-04-30T06:25:50.709ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74102025-04-30T06:25:50.709ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60972 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c5b65cf5-b7b3-4dab-b697-44b3d507b879
74112025-04-30T06:25:50.863ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
74122025-04-30T06:25:50.863ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
74132025-04-30T06:25:50.864ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ea c8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53675 req_id = 9f152af4-e9a5-4ca7-ba57-ee9393eae2b3
74142025-04-30T06:25:50.864ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60104,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53675 req_id = 9f152af4-e9a5-4ca7-ba57-ee9393eae2b3
74152025-04-30T06:25:50.864ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
74162025-04-30T06:25:50.864ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60104, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53675 req_id = 9f152af4-e9a5-4ca7-ba57-ee9393eae2b3
74172025-04-30T06:25:50.864ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
74182025-04-30T06:25:50.864ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
74192025-04-30T06:25:50.864ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
74202025-04-30T06:25:50.867ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b25329a-2d4e-4d9c-820d-3cd778488895", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74212025-04-30T06:25:50.867ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
74222025-04-30T06:25:50.872ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a63620d-d860-4488-a65b-0d91263cc490", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74232025-04-30T06:25:50.872ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
74242025-04-30T06:25:50.875ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab2498c-e7bc-413f-a140-468e6c0ce362", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74252025-04-30T06:25:50.875ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
74262025-04-30T06:25:50.877ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be33c274-426a-4462-ad1c-7733dad8e812", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74272025-04-30T06:25:50.877ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
74282025-04-30T06:25:50.892ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb4b93d6-a75a-4337-9f84-1c32fcc99e8c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74292025-04-30T06:25:50.893ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
74302025-04-30T06:25:50.896ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dafcf34e-37d3-4f18-968f-a3742bbf80cb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74312025-04-30T06:25:50.897ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
74322025-04-30T06:25:50.901ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc9cdae3-185c-476c-92c6-f64542b66c36", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74332025-04-30T06:25:50.901ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
74342025-04-30T06:25:50.915ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
74352025-04-30T06:25:50.918ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b271bd1-b428-47cc-bb8b-5d4fc933f043", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74362025-04-30T06:25:50.919ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
74372025-04-30T06:25:50.920ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76606c69-4d15-4a39-b970-dfad243b5f47", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74382025-04-30T06:25:50.920ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
74392025-04-30T06:25:50.921ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eea99f21-aa61-44df-a427-e89d7e9cc9f5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74402025-04-30T06:25:50.921ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
74412025-04-30T06:25:50.923ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af87eccd-808f-468d-9db6-e0cf2e32f82b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74422025-04-30T06:25:50.925ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
74432025-04-30T06:25:50.926ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ee2e270-99c6-4d1f-af1f-280a363af974", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74442025-04-30T06:25:50.927ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
74452025-04-30T06:25:50.928ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3449b5e-80a1-4f58-a34d-a96e2b4f228c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74462025-04-30T06:25:50.928ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
74472025-04-30T06:25:50.931ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cae7818-cbf9-40df-9a1d-827962a35e51", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74482025-04-30T06:25:50.931ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
74492025-04-30T06:25:50.933ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7ccad3f-646f-4dad-8b27-01ddec09543c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74502025-04-30T06:25:50.933ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
74512025-04-30T06:25:50.938ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f396c49-a4a1-4a85-834a-c2c6e7faebdc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74522025-04-30T06:25:50.938ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
74532025-04-30T06:25:50.941ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50c96ab6-4bcc-4607-bde7-a38280e3f273", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74542025-04-30T06:25:50.941ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
74552025-04-30T06:25:50.945ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01556038-6474-413f-aadf-26bbf9e6c156", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74562025-04-30T06:25:50.945ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
74572025-04-30T06:25:50.949ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1da6a07f-3804-4ca1-b190-da032a9a1ea6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74582025-04-30T06:25:50.949ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
74592025-04-30T06:25:50.950ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67dfedbb-565b-49da-aff4-91cd49ccb733", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74602025-04-30T06:25:50.950ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
74612025-04-30T06:25:50.951ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46b065ab-f3e5-4c6b-b533-1b87ccce36fd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74622025-04-30T06:25:50.951ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
74632025-04-30T06:25:50.953ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db270c29-fb98-4fa4-b9a3-0afca44928cb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74642025-04-30T06:25:50.953ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
74652025-04-30T06:25:50.955ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05a0dfb5-dafa-4a16-acc9-be2a85f479aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74662025-04-30T06:25:50.955ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
74672025-04-30T06:25:50.956ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "137ec082-fe93-43b7-abc6-4634872b5316", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74682025-04-30T06:25:50.956ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
74692025-04-30T06:25:50.959ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eefac7d0-6151-4a6b-934b-44614dcf1657", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74702025-04-30T06:25:50.959ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
74712025-04-30T06:25:50.972ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74722025-04-30T06:25:50.972ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74732025-04-30T06:25:50.973ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74742025-04-30T06:25:50.973ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
74752025-04-30T06:25:50.974ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74762025-04-30T06:25:50.978ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f84d3a-f774-4a81-9495-fedd3a89c26b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74772025-04-30T06:25:50.979ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
74782025-04-30T06:25:50.979ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74792025-04-30T06:25:50.989ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74802025-04-30T06:25:50.989ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74812025-04-30T06:25:50.990ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
74822025-04-30T06:25:50.990ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74832025-04-30T06:25:50.993ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c16bc6a-42a2-41b0-827f-116ccac7c54c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74842025-04-30T06:25:50.993ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
74852025-04-30T06:25:50.993ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74862025-04-30T06:25:50.994ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
74872025-04-30T06:25:50.995ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74882025-04-30T06:25:51.002ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e324cb3-e149-4843-bc4e-341c23f1d93f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74892025-04-30T06:25:51.002ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
74902025-04-30T06:25:51.007ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60ed68da-1be6-4cc9-a8d6-c5e914d0c239", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74912025-04-30T06:25:51.007ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
74922025-04-30T06:25:51.011ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f7a4858-6729-4872-873f-0dc729eae54a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74932025-04-30T06:25:51.011ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
74942025-04-30T06:25:51.013ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ca0cc06-b5eb-4613-8f82-79401890fe62", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74952025-04-30T06:25:51.014ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
74962025-04-30T06:25:51.015ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e69972-89ca-4082-897e-24ed89767025", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74972025-04-30T06:25:51.016ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
74982025-04-30T06:25:51.017ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af55c0e0-85cf-4b89-b46f-b737e473202f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
74992025-04-30T06:25:51.020ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
75002025-04-30T06:25:51.027ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c736b92-ef08-4f67-b299-f17f240cb38a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75012025-04-30T06:25:51.027ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
75022025-04-30T06:25:51.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdae634d-109a-49db-b12b-3960cd2d0b5d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75032025-04-30T06:25:51.030ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
75042025-04-30T06:25:51.032ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40b64c5-5df3-4ee4-a0cd-f79ffd06e757", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75052025-04-30T06:25:51.033ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
75062025-04-30T06:25:51.038ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c41768-e7be-48d2-a4af-4c5aadd7ed96", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75072025-04-30T06:25:51.038ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
75082025-04-30T06:25:51.038ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
75092025-04-30T06:25:51.043ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f1e788e-6bcf-4e17-8e8b-11dc9207ae70", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75102025-04-30T06:25:51.043ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
75112025-04-30T06:25:51.047ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f948f98-0b59-4a7c-b2c5-0d04a8db75ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75122025-04-30T06:25:51.047ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
75132025-04-30T06:25:51.052ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3bfafd8-2be2-4fec-a3ea-0cc2478f791d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75142025-04-30T06:25:51.052ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
75152025-04-30T06:25:51.055ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05b4b331-ee82-4359-b170-127aa91a2df1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75162025-04-30T06:25:51.055ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
75172025-04-30T06:25:51.056ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2073d6f-cbbd-462a-a41b-ee079173fdea", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75182025-04-30T06:25:51.056ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
75192025-04-30T06:25:51.058ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b02f6cd5-d92c-4f83-9f24-fa0222670ba8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75202025-04-30T06:25:51.058ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
75212025-04-30T06:25:51.059ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f8dcbc6-8551-4829-ac6a-2d392741d98a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75222025-04-30T06:25:51.059ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
75232025-04-30T06:25:51.060ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c343fdb-1840-42d0-bbef-c02575ca272f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75242025-04-30T06:25:51.060ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
75252025-04-30T06:25:51.060ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
75262025-04-30T06:25:51.062ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6e9386c-3820-403f-a376-729d812fbd3d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75272025-04-30T06:25:51.064ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
75282025-04-30T06:25:51.066ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7da96deb-e739-46ad-97ee-b1b51c47e6fb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75292025-04-30T06:25:51.066ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
75302025-04-30T06:25:51.069ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e499e92-bdf7-4d28-ae31-235f55e34ee9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75312025-04-30T06:25:51.070ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
75322025-04-30T06:25:51.079ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "32f8a146-0c62-42e0-9aa4-f172dabd56ea", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75332025-04-30T06:25:51.082ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
75342025-04-30T06:25:51.085ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f578d6ab-32ba-4507-80ba-b6913bf772d7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75352025-04-30T06:25:51.085ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75362025-04-30T06:25:51.085ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75372025-04-30T06:25:51.085ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75382025-04-30T06:25:51.086ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75392025-04-30T06:25:51.087ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75402025-04-30T06:25:51.091ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75412025-04-30T06:25:51.091ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75422025-04-30T06:25:51.091ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75432025-04-30T06:25:51.091ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
75442025-04-30T06:25:51.092ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75452025-04-30T06:25:51.092ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75462025-04-30T06:25:51.095ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75472025-04-30T06:25:51.096ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
75482025-04-30T06:25:51.096ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75492025-04-30T06:25:51.096ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75502025-04-30T06:25:51.096ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75512025-04-30T06:25:51.096ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75522025-04-30T06:25:51.110ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75532025-04-30T06:25:51.119ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75542025-04-30T06:25:51.119ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75552025-04-30T06:25:51.121ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
75562025-04-30T06:25:51.127ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75572025-04-30T06:25:51.130ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75582025-04-30T06:25:51.136ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
75592025-04-30T06:25:51.137ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
75602025-04-30T06:25:51.137ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
75612025-04-30T06:25:51.137ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
75622025-04-30T06:25:51.140ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75632025-04-30T06:25:51.140ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
75642025-04-30T06:25:51.140ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75652025-04-30T06:25:51.249ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
75662025-04-30T06:25:51.249ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
75672025-04-30T06:25:51.249ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:39825 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
75682025-04-30T06:25:51.253ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:39825 req_id = 50eee552-848f-4af6-a3dc-7d17ad463f3e sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
75692025-04-30T06:25:51.253ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 615 local_addr = [::1]:42668 method = GET remote_addr = [::1]:39825 req_id = 50eee552-848f-4af6-a3dc-7d17ad463f3e response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
75702025-04-30T06:25:51.253ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "50eee552-848f-4af6-a3dc-7d17ad463f3e", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75712025-04-30T06:25:51.366ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
75722025-04-30T06:25:51.366ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
75732025-04-30T06:25:51.369ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:64858 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
75742025-04-30T06:25:51.369ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
75752025-04-30T06:25:51.369ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75762025-04-30T06:25:51.369ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 965ef809-2ecd-4766-94a9-48313356d033 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75772025-04-30T06:25:51.369ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:64858 req_id = 38268a0c-252a-4138-9d18-e5f051727745 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
75782025-04-30T06:25:51.376ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 965ef809-2ecd-4766-94a9-48313356d033 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75792025-04-30T06:25:51.376ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 965ef809-2ecd-4766-94a9-48313356d033 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75802025-04-30T06:25:51.388ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 965ef809-2ecd-4766-94a9-48313356d033 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75812025-04-30T06:25:51.392ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 23633 local_addr = [::1]:47918 method = GET remote_addr = [::1]:64858 req_id = 38268a0c-252a-4138-9d18-e5f051727745 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
75822025-04-30T06:25:51.394ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "38268a0c-252a-4138-9d18-e5f051727745", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75832025-04-30T06:25:51.568ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
75842025-04-30T06:25:51.569ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
75852025-04-30T06:25:51.570ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:34743 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
75862025-04-30T06:25:51.571ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75872025-04-30T06:25:51.576ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75882025-04-30T06:25:51.577ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 208585 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 965ef809-2ecd-4766-94a9-48313356d033 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
75892025-04-30T06:25:51.581ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:34743 req_id = d0721c86-551b-4b6b-b21b-c886c4089bb1 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
75902025-04-30T06:25:51.581ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "965ef809-2ecd-4766-94a9-48313356d033", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
75912025-04-30T06:25:51.581ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
75922025-04-30T06:25:51.582ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1622 local_addr = [::1]:35744 method = GET remote_addr = [::1]:34743 req_id = d0721c86-551b-4b6b-b21b-c886c4089bb1 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
75932025-04-30T06:25:51.588ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 773700f3-111a-48bc-ac58-6c07a9509cd8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
75942025-04-30T06:25:51.595ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 773700f3-111a-48bc-ac58-6c07a9509cd8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
75952025-04-30T06:25:51.595ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 773700f3-111a-48bc-ac58-6c07a9509cd8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
75962025-04-30T06:25:51.596ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 773700f3-111a-48bc-ac58-6c07a9509cd8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
75972025-04-30T06:25:51.596ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75982025-04-30T06:25:51.598ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
75992025-04-30T06:25:51.598ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76002025-04-30T06:25:51.598ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76012025-04-30T06:25:51.598ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0721c86-551b-4b6b-b21b-c886c4089bb1", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
76022025-04-30T06:25:51.671ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
76032025-04-30T06:25:51.671ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
76042025-04-30T06:25:51.674ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:43544 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
76052025-04-30T06:25:51.674ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76062025-04-30T06:25:51.674ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76072025-04-30T06:25:51.674ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
76082025-04-30T06:25:51.675ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
76092025-04-30T06:25:51.679ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:43544 req_id = 00e01b1e-910e-4d47-8d42-59cb45418d92 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76102025-04-30T06:25:51.689ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 9171 local_addr = [::1]:35343 method = GET remote_addr = [::1]:43544 req_id = 00e01b1e-910e-4d47-8d42-59cb45418d92 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76112025-04-30T06:25:51.692ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
76122025-04-30T06:25:51.692ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
76132025-04-30T06:25:51.693ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "00e01b1e-910e-4d47-8d42-59cb45418d92", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
76142025-04-30T06:25:51.697ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76152025-04-30T06:25:51.697ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76162025-04-30T06:25:51.698ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
76172025-04-30T06:25:51.711ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76182025-04-30T06:25:51.711ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76192025-04-30T06:25:51.711ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76202025-04-30T06:25:51.711ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76212025-04-30T06:25:51.712ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76222025-04-30T06:25:51.715ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 126558 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 773700f3-111a-48bc-ac58-6c07a9509cd8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
76232025-04-30T06:25:51.731ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "773700f3-111a-48bc-ac58-6c07a9509cd8", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:50 GMT"} })
76242025-04-30T06:25:51.732ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76252025-04-30T06:25:51.732ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
76262025-04-30T06:25:51.732ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
76272025-04-30T06:25:51.732ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76282025-04-30T06:25:51.737ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76292025-04-30T06:25:51.737ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76302025-04-30T06:25:51.737ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76312025-04-30T06:25:51.738ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76322025-04-30T06:25:51.739ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
76332025-04-30T06:25:51.739ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76342025-04-30T06:25:51.768ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
76352025-04-30T06:25:51.775ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
76362025-04-30T06:25:51.794ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
76372025-04-30T06:25:51.795ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
76382025-04-30T06:25:51.795ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.141800893s iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
76392025-04-30T06:25:51.795ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
76402025-04-30T06:25:51.795ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76412025-04-30T06:25:51.796ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76422025-04-30T06:25:51.796ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76432025-04-30T06:25:51.796ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76442025-04-30T06:25:51.797ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76452025-04-30T06:25:51.801ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76462025-04-30T06:25:51.811ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76472025-04-30T06:25:51.811ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76482025-04-30T06:25:51.811ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76492025-04-30T06:25:51.812ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76502025-04-30T06:25:51.822ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76512025-04-30T06:25:51.822ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76522025-04-30T06:25:51.822ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76532025-04-30T06:25:51.822ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
76542025-04-30T06:25:51.824ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76552025-04-30T06:25:51.824ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
76562025-04-30T06:25:51.825ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
76572025-04-30T06:25:51.826ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 1a 9e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49168 req_id = fe9a1c70-85e2-4aa2-82d5-f3d53d6b3c88
76582025-04-30T06:25:51.826ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6814,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49168 req_id = fe9a1c70-85e2-4aa2-82d5-f3d53d6b3c88
76592025-04-30T06:25:51.827ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
76602025-04-30T06:25:51.828ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
76612025-04-30T06:25:51.828ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76622025-04-30T06:25:51.828ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49168 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fe9a1c70-85e2-4aa2-82d5-f3d53d6b3c88
76632025-04-30T06:25:51.829ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3216, tv_nsec: 628594062 } })
76642025-04-30T06:25:51.829ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
76652025-04-30T06:25:51.829ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
76662025-04-30T06:25:51.830ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e0 3d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .=..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53068 req_id = 3dc30e39-e779-487d-ae31-d8b40e2b5729
76672025-04-30T06:25:51.830ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57405,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53068 req_id = 3dc30e39-e779-487d-ae31-d8b40e2b5729
76682025-04-30T06:25:51.830ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
76692025-04-30T06:25:51.830ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53068 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3dc30e39-e779-487d-ae31-d8b40e2b5729
76702025-04-30T06:25:51.830ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c2 5c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .\\..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53341 req_id = 6b644356-39b6-49de-96ea-872e4ef3b605
76712025-04-30T06:25:51.830ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49756,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53341 req_id = 6b644356-39b6-49de-96ea-872e4ef3b605
76722025-04-30T06:25:51.831ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76732025-04-30T06:25:51.831ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53341 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6b644356-39b6-49de-96ea-872e4ef3b605
76742025-04-30T06:25:51.831ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76752025-04-30T06:25:51.839ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76762025-04-30T06:25:51.839ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76772025-04-30T06:25:51.839ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
76782025-04-30T06:25:51.840ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76792025-04-30T06:25:51.945ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
76802025-04-30T06:25:51.948ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9b cb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55042 req_id = 3aa31028-bf89-4b7a-b758-535e6a73b193
76812025-04-30T06:25:51.948ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39883,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55042 req_id = 3aa31028-bf89-4b7a-b758-535e6a73b193
76822025-04-30T06:25:51.948ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
76832025-04-30T06:25:51.948ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39883, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55042 req_id = 3aa31028-bf89-4b7a-b758-535e6a73b193
76842025-04-30T06:25:51.949ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
76852025-04-30T06:25:51.949ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
76862025-04-30T06:25:51.949ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
76872025-04-30T06:25:51.950ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76882025-04-30T06:25:51.950ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
76892025-04-30T06:25:51.951ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76902025-04-30T06:25:51.954ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a88c545-6661-4581-aa49-9eea8efa2b32", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
76912025-04-30T06:25:51.954ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
76922025-04-30T06:25:51.957ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9935e25c-6454-47b0-b287-e2392b91f853", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
76932025-04-30T06:25:51.957ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
76942025-04-30T06:25:51.962ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62564198-e08a-44af-a776-98a088c5620d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
76952025-04-30T06:25:51.962ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
76962025-04-30T06:25:51.965ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "820d6472-92af-4c84-8083-64f3b965ba8f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
76972025-04-30T06:25:51.965ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
76982025-04-30T06:25:51.974ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a88099c1-c8ce-447c-bf00-4eb9884ab7aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
76992025-04-30T06:25:51.975ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
77002025-04-30T06:25:51.980ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd3f8968-4e47-4ec9-b4ed-5fbc920fa860", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77012025-04-30T06:25:51.980ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
77022025-04-30T06:25:51.982ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e68e9cdf-9b4f-494d-b225-2847c9d1cae7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77032025-04-30T06:25:51.983ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
77042025-04-30T06:25:51.985ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "633eefa2-637b-43b0-bf28-2bec81b4273f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77052025-04-30T06:25:51.985ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
77062025-04-30T06:25:51.987ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eed60e32-e3e3-4c74-8fad-ab37a23977dc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77072025-04-30T06:25:51.987ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
77082025-04-30T06:25:51.988ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c354788-1e86-4caf-92bd-5506de3c572e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77092025-04-30T06:25:51.989ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
77102025-04-30T06:25:51.991ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c318b9b2-b30b-4650-8662-45268ce47d12", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77112025-04-30T06:25:51.992ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
77122025-04-30T06:25:51.992ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
77132025-04-30T06:25:51.995ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e40d475-fcb1-4df9-8967-9ca07f0aa642", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77142025-04-30T06:25:51.995ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
77152025-04-30T06:25:51.996ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1137aef-4dae-40d1-abb1-bc4c25ef9902", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77162025-04-30T06:25:51.996ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
77172025-04-30T06:25:51.999ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73749726-aaed-47ad-938d-1deb9f4d9cff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77182025-04-30T06:25:51.999ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
77192025-04-30T06:25:52.004ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86a885fa-c20f-40f5-bba4-941f2ffdcfd9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77202025-04-30T06:25:52.005ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
77212025-04-30T06:25:52.007ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c89d3180-97eb-4d59-8d9e-ab8710e7b8c8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77222025-04-30T06:25:52.007ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
77232025-04-30T06:25:52.010ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8122f637-2b0f-482b-a907-01306d83f341", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77242025-04-30T06:25:52.010ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
77252025-04-30T06:25:52.012ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce4e56f6-5b1c-4a39-8e88-d62e2fbe1dae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77262025-04-30T06:25:52.012ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
77272025-04-30T06:25:52.015ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe86e0be-3d96-4d2c-a55d-6946a825e80d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77282025-04-30T06:25:52.015ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
77292025-04-30T06:25:52.016ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aaa69dd-7c37-4632-97e9-fa81b70b4fe7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77302025-04-30T06:25:52.016ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
77312025-04-30T06:25:52.017ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e462799-6912-4f75-9769-d4a62a8b0df2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77322025-04-30T06:25:52.018ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
77332025-04-30T06:25:52.019ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4340f61b-dec3-40b9-a85d-27afc10be1f6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77342025-04-30T06:25:52.019ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
77352025-04-30T06:25:52.020ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf6f8cc9-b02f-4ca2-8591-f45d4e0300bb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77362025-04-30T06:25:52.021ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
77372025-04-30T06:25:52.022ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea627a92-2643-49dc-93a0-b74acc4ce55e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77382025-04-30T06:25:52.022ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
77392025-04-30T06:25:52.024ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16db41e5-0fdf-4fda-a12b-ea23e3f81cfb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77402025-04-30T06:25:52.024ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
77412025-04-30T06:25:52.026ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f51cfab-ce27-4911-a499-9c6987b122db", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77422025-04-30T06:25:52.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
77432025-04-30T06:25:52.028ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8097d7e9-f565-4df9-89fd-c1edb2d784ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77442025-04-30T06:25:52.028ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
77452025-04-30T06:25:52.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "36b9e1fc-6cdc-4ade-9a31-aae1e098665d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77462025-04-30T06:25:52.030ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
77472025-04-30T06:25:52.034ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e76f70-1678-4509-9b64-bac492f314be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77482025-04-30T06:25:52.034ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
77492025-04-30T06:25:52.036ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0edc41b-d18a-4f9c-8ef8-9cff2d3f3772", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77502025-04-30T06:25:52.036ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
77512025-04-30T06:25:52.037ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e94bcb18-1566-4dbe-92b8-72ce811fd87d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77522025-04-30T06:25:52.037ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
77532025-04-30T06:25:52.038ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7459524e-143b-4e9f-9239-4fa7dc7b53bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77542025-04-30T06:25:52.038ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
77552025-04-30T06:25:52.039ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfa15895-c283-467e-aa05-2dc38aae58b4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77562025-04-30T06:25:52.039ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
77572025-04-30T06:25:52.040ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fc7445b-aaca-496a-a393-b2a721f683ba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77582025-04-30T06:25:52.040ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
77592025-04-30T06:25:52.041ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d54ed9d-f953-4f4f-990a-eb61fc72e02a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77602025-04-30T06:25:52.041ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
77612025-04-30T06:25:52.043ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3b4f8de-c6f6-4dee-a8e8-3ad39d0f1ce1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77622025-04-30T06:25:52.043ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
77632025-04-30T06:25:52.044ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77642025-04-30T06:25:52.044ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77652025-04-30T06:25:52.044ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77662025-04-30T06:25:52.044ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
77672025-04-30T06:25:52.046ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77682025-04-30T06:25:52.048ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43bfdf3c-3c82-48c7-8991-294e97f1219a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77692025-04-30T06:25:52.048ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
77702025-04-30T06:25:52.048ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77712025-04-30T06:25:52.055ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77722025-04-30T06:25:52.055ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77732025-04-30T06:25:52.055ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
77742025-04-30T06:25:52.056ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77752025-04-30T06:25:52.058ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "333ea858-9681-4327-be25-1d35fedd93d7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77762025-04-30T06:25:52.058ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
77772025-04-30T06:25:52.059ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77782025-04-30T06:25:52.059ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
77792025-04-30T06:25:52.059ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77802025-04-30T06:25:52.063ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1862ef1-f2b9-42cc-b36c-347855424669", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77812025-04-30T06:25:52.063ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
77822025-04-30T06:25:52.065ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67d71c4b-e4cb-4513-8898-bdc30d3c2589", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77832025-04-30T06:25:52.067ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
77842025-04-30T06:25:52.070ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10aa0e4b-e7d6-448b-ae89-bbc7b1c81480", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77852025-04-30T06:25:52.070ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
77862025-04-30T06:25:52.071ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10f4360c-11ab-4378-ae02-c09addcd2b61", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77872025-04-30T06:25:52.071ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
77882025-04-30T06:25:52.072ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b925c4c7-59aa-4133-af96-9b113251ca42", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77892025-04-30T06:25:52.072ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
77902025-04-30T06:25:52.074ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c3341c5-5546-4689-b6bd-9ce08fb1876e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77912025-04-30T06:25:52.074ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
77922025-04-30T06:25:52.075ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea94e929-313a-451a-8a96-b0703ced01aa", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77932025-04-30T06:25:52.075ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
77942025-04-30T06:25:52.075ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
77952025-04-30T06:25:52.082ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7922df0a-e8d1-44fb-bc29-87bcf3c003cd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77962025-04-30T06:25:52.083ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
77972025-04-30T06:25:52.085ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce04b39c-0da9-436f-bccb-63801fb9589c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
77982025-04-30T06:25:52.086ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
77992025-04-30T06:25:52.092ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c956885-cc7d-4e6c-a8fd-48161e3f2db0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78002025-04-30T06:25:52.093ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
78012025-04-30T06:25:52.096ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d803e82-5b73-4bea-89be-68dc5efb7789", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78022025-04-30T06:25:52.097ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
78032025-04-30T06:25:52.098ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1d67d29-736a-4a58-a393-3b01e976cc01", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78042025-04-30T06:25:52.099ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78052025-04-30T06:25:52.099ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78062025-04-30T06:25:52.099ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78072025-04-30T06:25:52.099ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78082025-04-30T06:25:52.107ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78092025-04-30T06:25:52.117ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78102025-04-30T06:25:52.124ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78112025-04-30T06:25:52.124ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78122025-04-30T06:25:52.124ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78132025-04-30T06:25:52.124ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78142025-04-30T06:25:52.124ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78152025-04-30T06:25:52.125ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78162025-04-30T06:25:52.126ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
78172025-04-30T06:25:52.130ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78182025-04-30T06:25:52.137ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78192025-04-30T06:25:52.138ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78202025-04-30T06:25:52.138ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78212025-04-30T06:25:52.139ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78222025-04-30T06:25:52.208ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
78232025-04-30T06:25:52.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
78242025-04-30T06:25:52.213ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78252025-04-30T06:25:52.213ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78262025-04-30T06:25:52.213ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78272025-04-30T06:25:52.213ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
78282025-04-30T06:25:52.214ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78292025-04-30T06:25:52.215ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:45347 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
78302025-04-30T06:25:52.215ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:45347 req_id = 02c49ea4-da90-466b-a69d-beab85ec1860 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
78312025-04-30T06:25:52.220ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 4443 local_addr = [::1]:42668 method = GET remote_addr = [::1]:45347 req_id = 02c49ea4-da90-466b-a69d-beab85ec1860 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
78322025-04-30T06:25:52.221ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c49ea4-da90-466b-a69d-beab85ec1860", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78332025-04-30T06:25:52.308ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
78342025-04-30T06:25:52.308ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
78352025-04-30T06:25:52.309ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78362025-04-30T06:25:52.314ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78372025-04-30T06:25:52.314ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78382025-04-30T06:25:52.314ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
78392025-04-30T06:25:52.314ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78402025-04-30T06:25:52.314ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:49825 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
78412025-04-30T06:25:52.319ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:49825 req_id = c053799f-edf8-4f5c-ada9-1a6dc6166265 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
78422025-04-30T06:25:52.319ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 279 local_addr = [::1]:47918 method = GET remote_addr = [::1]:49825 req_id = c053799f-edf8-4f5c-ada9-1a6dc6166265 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
78432025-04-30T06:25:52.321ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c053799f-edf8-4f5c-ada9-1a6dc6166265", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78442025-04-30T06:25:52.358ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
78452025-04-30T06:25:52.358ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
78462025-04-30T06:25:52.358ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78472025-04-30T06:25:52.358ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
78482025-04-30T06:25:52.359ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78492025-04-30T06:25:52.360ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:61461 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
78502025-04-30T06:25:52.361ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:61461 req_id = a2a3dada-9c4a-4b7d-b528-c9bbf659278c sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
78512025-04-30T06:25:52.363ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2187 local_addr = [::1]:35744 method = GET remote_addr = [::1]:61461 req_id = a2a3dada-9c4a-4b7d-b528-c9bbf659278c response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
78522025-04-30T06:25:52.363ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2a3dada-9c4a-4b7d-b528-c9bbf659278c", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78532025-04-30T06:25:52.416ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
78542025-04-30T06:25:52.416ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
78552025-04-30T06:25:52.417ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:33430 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
78562025-04-30T06:25:52.417ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78572025-04-30T06:25:52.419ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78582025-04-30T06:25:52.420ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:33430 req_id = 0a1c7f68-aa81-45b6-946b-73217768e24e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
78592025-04-30T06:25:52.422ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1791 local_addr = [::1]:35343 method = GET remote_addr = [::1]:33430 req_id = 0a1c7f68-aa81-45b6-946b-73217768e24e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
78602025-04-30T06:25:52.426ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78612025-04-30T06:25:52.427ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78622025-04-30T06:25:52.427ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78632025-04-30T06:25:52.427ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
78642025-04-30T06:25:52.427ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a1c7f68-aa81-45b6-946b-73217768e24e", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
78652025-04-30T06:25:52.430ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
78662025-04-30T06:25:52.430ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
78672025-04-30T06:25:52.430ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
78682025-04-30T06:25:52.436ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
78692025-04-30T06:25:52.436ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78702025-04-30T06:25:52.436ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78712025-04-30T06:25:52.436ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78722025-04-30T06:25:52.437ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78732025-04-30T06:25:52.443ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78742025-04-30T06:25:52.454ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78752025-04-30T06:25:52.454ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78762025-04-30T06:25:52.455ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78772025-04-30T06:25:52.455ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78782025-04-30T06:25:52.494ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
78792025-04-30T06:25:52.500ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
78802025-04-30T06:25:52.502ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
78812025-04-30T06:25:52.502ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 707.002657ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
78822025-04-30T06:25:52.502ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
78832025-04-30T06:25:52.502ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
78842025-04-30T06:25:52.502ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78852025-04-30T06:25:52.502ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78862025-04-30T06:25:52.502ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78872025-04-30T06:25:52.504ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78882025-04-30T06:25:52.506ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78892025-04-30T06:25:52.511ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78902025-04-30T06:25:52.511ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78912025-04-30T06:25:52.511ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78922025-04-30T06:25:52.512ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78932025-04-30T06:25:52.516ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
78942025-04-30T06:25:52.516ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
78952025-04-30T06:25:52.518ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 16 6e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .n..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58565 req_id = 80bcbb58-eec2-4b8e-92e4-86d187baa091
78962025-04-30T06:25:52.519ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5742,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58565 req_id = 80bcbb58-eec2-4b8e-92e4-86d187baa091
78972025-04-30T06:25:52.519ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
78982025-04-30T06:25:52.520ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
78992025-04-30T06:25:52.520ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79002025-04-30T06:25:52.520ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58565 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 80bcbb58-eec2-4b8e-92e4-86d187baa091
79012025-04-30T06:25:52.521ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3217, tv_nsec: 320110884 } })
79022025-04-30T06:25:52.521ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
79032025-04-30T06:25:52.521ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
79042025-04-30T06:25:52.523ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: cc 2b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .+..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64474 req_id = 557e5775-73e3-4a21-a93f-ba17d2cfe6b4
79052025-04-30T06:25:52.523ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52267,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64474 req_id = 557e5775-73e3-4a21-a93f-ba17d2cfe6b4
79062025-04-30T06:25:52.523ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
79072025-04-30T06:25:52.523ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64474 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 557e5775-73e3-4a21-a93f-ba17d2cfe6b4
79082025-04-30T06:25:52.523ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d6 60 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .`..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52819 req_id = aab9a00a-5e6b-4649-bad3-43d922d651ad
79092025-04-30T06:25:52.523ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54880,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52819 req_id = aab9a00a-5e6b-4649-bad3-43d922d651ad
79102025-04-30T06:25:52.523ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
79112025-04-30T06:25:52.524ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52819 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = aab9a00a-5e6b-4649-bad3-43d922d651ad
79122025-04-30T06:25:52.670ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
79132025-04-30T06:25:52.673ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79142025-04-30T06:25:52.673ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79152025-04-30T06:25:52.673ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79162025-04-30T06:25:52.673ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
79172025-04-30T06:25:52.674ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79182025-04-30T06:25:52.674ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
79192025-04-30T06:25:52.674ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
79202025-04-30T06:25:52.675ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
79212025-04-30T06:25:52.676ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
79222025-04-30T06:25:52.677ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 20 17 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51318 req_id = 1b8fa43c-b728-4365-9eb8-7156d1863536
79232025-04-30T06:25:52.677ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8215,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51318 req_id = 1b8fa43c-b728-4365-9eb8-7156d1863536
79242025-04-30T06:25:52.677ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
79252025-04-30T06:25:52.678ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8215, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51318 req_id = 1b8fa43c-b728-4365-9eb8-7156d1863536
79262025-04-30T06:25:52.679ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
79272025-04-30T06:25:52.679ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
79282025-04-30T06:25:52.679ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
79292025-04-30T06:25:52.679ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79302025-04-30T06:25:52.687ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79312025-04-30T06:25:52.687ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79322025-04-30T06:25:52.687ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
79332025-04-30T06:25:52.689ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79342025-04-30T06:25:52.690ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
79352025-04-30T06:25:52.690ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
79362025-04-30T06:25:52.694ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79372025-04-30T06:25:52.694ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
79382025-04-30T06:25:52.695ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79392025-04-30T06:25:52.696ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
79402025-04-30T06:25:52.696ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
79412025-04-30T06:25:52.697ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7eabbd52-0c0e-451d-b593-5509d445acb4", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79422025-04-30T06:25:52.697ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
79432025-04-30T06:25:52.700ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca3b6d78-1f28-4cd6-927f-589cdcfbafbe", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79442025-04-30T06:25:52.700ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
79452025-04-30T06:25:52.705ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b79141e8-cd24-4459-90aa-8101f3e531ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79462025-04-30T06:25:52.706ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
79472025-04-30T06:25:52.714ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6e21d5c-c1bd-41b0-8d58-82451a360d02", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79482025-04-30T06:25:52.714ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
79492025-04-30T06:25:52.717ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "460aa8de-59b5-4205-8a94-5906df053f75", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79502025-04-30T06:25:52.717ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
79512025-04-30T06:25:52.719ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7ef8acb-8b8b-4530-bf9a-2daf9c65183f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79522025-04-30T06:25:52.719ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
79532025-04-30T06:25:52.723ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3745fa4-a564-434a-8cbf-e6ac74d42985", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79542025-04-30T06:25:52.724ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
79552025-04-30T06:25:52.732ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f82aa1f-de20-43ee-91d1-b7cfae280c4c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79562025-04-30T06:25:52.732ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
79572025-04-30T06:25:52.733ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
79582025-04-30T06:25:52.734ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "510dbaa7-5b10-47ae-a7eb-4e951010b457", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79592025-04-30T06:25:52.734ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
79602025-04-30T06:25:52.736ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b391bde-5316-4483-b456-514f600f1589", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79612025-04-30T06:25:52.736ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
79622025-04-30T06:25:52.737ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d81dbbe2-6aff-4bd6-8203-744c60d79668", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79632025-04-30T06:25:52.737ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
79642025-04-30T06:25:52.739ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "199db148-2637-4696-a518-bf427c46403d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79652025-04-30T06:25:52.739ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
79662025-04-30T06:25:52.740ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd37ee32-5e15-4efa-9a41-193570b5eca5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79672025-04-30T06:25:52.740ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
79682025-04-30T06:25:52.743ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5227ba4f-c646-433b-836f-a9674b3adb8f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79692025-04-30T06:25:52.743ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
79702025-04-30T06:25:52.746ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b23db4d-35bd-48ff-8f9f-22ae81db99c9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79712025-04-30T06:25:52.746ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
79722025-04-30T06:25:52.748ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82788936-cda0-4d38-a934-e85b23b02ebd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79732025-04-30T06:25:52.748ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
79742025-04-30T06:25:52.750ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2394b331-d3da-444b-919e-bd0425321968", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79752025-04-30T06:25:52.750ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
79762025-04-30T06:25:52.754ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b301f2a-dc11-41a1-98f7-037624ac18aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79772025-04-30T06:25:52.754ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
79782025-04-30T06:25:52.756ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52d73b0d-27ea-4561-ae0e-8b85fc4dfb5d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79792025-04-30T06:25:52.757ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
79802025-04-30T06:25:52.759ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7dea65a-6e50-49b3-94a4-47ded5b50dc0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79812025-04-30T06:25:52.759ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
79822025-04-30T06:25:52.761ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "289dba5a-2450-4832-b603-ea88c54f7e92", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79832025-04-30T06:25:52.761ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
79842025-04-30T06:25:52.763ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c3b01c7-975c-4697-b1bc-32619e7e160e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79852025-04-30T06:25:52.763ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
79862025-04-30T06:25:52.764ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba3f1864-8ed8-4f5d-bb92-d53382675082", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79872025-04-30T06:25:52.764ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
79882025-04-30T06:25:52.766ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ded61303-5afe-417c-a2d1-b2f9b77c099f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79892025-04-30T06:25:52.766ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
79902025-04-30T06:25:52.768ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "535450af-a2f0-4ccc-8c13-0e0bbf6cffbf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79912025-04-30T06:25:52.768ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
79922025-04-30T06:25:52.771ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de2a99a0-48c8-4ce3-875e-69f3620c1e49", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79932025-04-30T06:25:52.771ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
79942025-04-30T06:25:52.773ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37fdd512-784c-44aa-b0aa-7ada6c244f97", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79952025-04-30T06:25:52.773ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
79962025-04-30T06:25:52.775ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a461d2f4-ab5b-46d9-82bc-cdaac6df86ab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
79972025-04-30T06:25:52.775ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
79982025-04-30T06:25:52.776ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
79992025-04-30T06:25:52.778ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bd9e512-8f6b-4d76-9198-a7a6b279dd03", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80002025-04-30T06:25:52.778ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
80012025-04-30T06:25:52.780ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "179a8ed4-90f0-402b-bb62-aecaeee2650c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80022025-04-30T06:25:52.780ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
80032025-04-30T06:25:52.781ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ea7f0b-bc08-4321-8eb0-04002a375bd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80042025-04-30T06:25:52.782ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
80052025-04-30T06:25:52.783ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fecbe520-c6fe-4979-86a4-cc873cbb0666", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80062025-04-30T06:25:52.783ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
80072025-04-30T06:25:52.784ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80082025-04-30T06:25:52.784ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80092025-04-30T06:25:52.784ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80102025-04-30T06:25:52.784ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
80112025-04-30T06:25:52.785ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80122025-04-30T06:25:52.786ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93d641e8-eb9f-43f9-88dc-59bc2283d6d0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80132025-04-30T06:25:52.787ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
80142025-04-30T06:25:52.788ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80152025-04-30T06:25:52.793ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80162025-04-30T06:25:52.794ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80172025-04-30T06:25:52.794ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
80182025-04-30T06:25:52.794ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80192025-04-30T06:25:52.796ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d339b245-07f1-4e47-9b69-ba3f603fc6f7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80202025-04-30T06:25:52.796ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
80212025-04-30T06:25:52.797ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80222025-04-30T06:25:52.797ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
80232025-04-30T06:25:52.797ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80242025-04-30T06:25:52.800ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d1fa409-a407-4365-9324-ad056df3c2f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80252025-04-30T06:25:52.800ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
80262025-04-30T06:25:52.803ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6100e88a-1ed7-44b1-aabb-8975c791f6b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80272025-04-30T06:25:52.804ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
80282025-04-30T06:25:52.808ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b2d60e4-d159-419d-9842-eb031d1551e4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80292025-04-30T06:25:52.808ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
80302025-04-30T06:25:52.811ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c5a8156-7c6e-4288-ab38-2791c3b7101d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80312025-04-30T06:25:52.811ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
80322025-04-30T06:25:52.813ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6661deb-c60d-4fb1-a416-0b182ba43333", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80332025-04-30T06:25:52.813ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
80342025-04-30T06:25:52.816ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "884fc1c2-e044-4981-87dd-ac14a8ba2df9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80352025-04-30T06:25:52.816ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
80362025-04-30T06:25:52.818ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7b67372-9579-4b02-a269-6897e7bf829a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80372025-04-30T06:25:52.819ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
80382025-04-30T06:25:52.820ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0240f9e-6a04-4d25-9db9-d45a576b359e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80392025-04-30T06:25:52.820ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
80402025-04-30T06:25:52.826ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29b6821a-fdb6-45f5-864e-d2c4ffd7248a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80412025-04-30T06:25:52.827ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
80422025-04-30T06:25:52.828ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
80432025-04-30T06:25:52.830ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37dad3d8-c18e-463d-880c-579d7b10d9c9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80442025-04-30T06:25:52.830ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
80452025-04-30T06:25:52.831ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7924ce7-3f48-4333-ae6c-2b82eeab850d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80462025-04-30T06:25:52.832ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
80472025-04-30T06:25:52.832ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
80482025-04-30T06:25:52.833ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "42579676-9ad0-494c-93c8-0d89494da716", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80492025-04-30T06:25:52.833ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
80502025-04-30T06:25:52.835ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2cd81ad-2906-4da7-ab66-5119dc5b3a19", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80512025-04-30T06:25:52.835ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
80522025-04-30T06:25:52.836ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "05d5861e-2049-4dc2-b4d4-b69d657b3d8e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80532025-04-30T06:25:52.837ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
80542025-04-30T06:25:52.839ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e62b847-e7f4-4c09-b294-e1052bdd812b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80552025-04-30T06:25:52.839ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
80562025-04-30T06:25:52.839ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25c43df5-54bc-470a-b3bd-fa9424d449cc", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80572025-04-30T06:25:52.839ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80582025-04-30T06:25:52.839ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80592025-04-30T06:25:52.839ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80602025-04-30T06:25:52.839ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80612025-04-30T06:25:52.840ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80622025-04-30T06:25:52.844ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80632025-04-30T06:25:52.852ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80642025-04-30T06:25:52.853ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80652025-04-30T06:25:52.853ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80662025-04-30T06:25:52.853ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80672025-04-30T06:25:52.854ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80682025-04-30T06:25:52.854ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80692025-04-30T06:25:52.859ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80702025-04-30T06:25:52.866ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80712025-04-30T06:25:52.866ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80722025-04-30T06:25:52.867ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80732025-04-30T06:25:52.867ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80742025-04-30T06:25:52.926ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
80752025-04-30T06:25:52.926ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
80762025-04-30T06:25:52.927ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80772025-04-30T06:25:52.927ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80782025-04-30T06:25:52.927ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80792025-04-30T06:25:52.927ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
80802025-04-30T06:25:52.928ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80812025-04-30T06:25:52.929ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:32785 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
80822025-04-30T06:25:52.930ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:32785 req_id = a70b0833-a315-4e46-a3f8-427aa769fc85 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
80832025-04-30T06:25:52.930ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 427 local_addr = [::1]:42668 method = GET remote_addr = [::1]:32785 req_id = a70b0833-a315-4e46-a3f8-427aa769fc85 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
80842025-04-30T06:25:52.930ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a70b0833-a315-4e46-a3f8-427aa769fc85", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:51 GMT"} })
80852025-04-30T06:25:52.992ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
80862025-04-30T06:25:52.992ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
80872025-04-30T06:25:52.993ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80882025-04-30T06:25:52.998ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80892025-04-30T06:25:52.998ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80902025-04-30T06:25:52.998ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
80912025-04-30T06:25:52.998ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80922025-04-30T06:25:52.998ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:49984 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
80932025-04-30T06:25:52.999ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:49984 req_id = b1b03267-b3b2-49f5-9082-86d73e297b6a sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
80942025-04-30T06:25:53.001ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1238 local_addr = [::1]:47918 method = GET remote_addr = [::1]:49984 req_id = b1b03267-b3b2-49f5-9082-86d73e297b6a response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
80952025-04-30T06:25:53.001ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1b03267-b3b2-49f5-9082-86d73e297b6a", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
80962025-04-30T06:25:53.073ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
80972025-04-30T06:25:53.073ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
80982025-04-30T06:25:53.074ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80992025-04-30T06:25:53.074ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
81002025-04-30T06:25:53.074ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81012025-04-30T06:25:53.075ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:63386 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
81022025-04-30T06:25:53.075ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:63386 req_id = c5be5a89-ebe7-49d9-b448-e75923072496 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
81032025-04-30T06:25:53.075ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 252 local_addr = [::1]:35744 method = GET remote_addr = [::1]:63386 req_id = c5be5a89-ebe7-49d9-b448-e75923072496 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
81042025-04-30T06:25:53.076ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5be5a89-ebe7-49d9-b448-e75923072496", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
81052025-04-30T06:25:53.124ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
81062025-04-30T06:25:53.124ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
81072025-04-30T06:25:53.125ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:41196 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
81082025-04-30T06:25:53.126ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:41196 req_id = 7d2c0b58-ac36-4b92-8413-d6279a3cf1c3 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81092025-04-30T06:25:53.126ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 593 local_addr = [::1]:35343 method = GET remote_addr = [::1]:41196 req_id = 7d2c0b58-ac36-4b92-8413-d6279a3cf1c3 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
81102025-04-30T06:25:53.127ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2c0b58-ac36-4b92-8413-d6279a3cf1c3", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
81112025-04-30T06:25:53.129ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81122025-04-30T06:25:53.129ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
81132025-04-30T06:25:53.129ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
81142025-04-30T06:25:53.133ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81152025-04-30T06:25:53.134ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81162025-04-30T06:25:53.134ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81172025-04-30T06:25:53.134ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81182025-04-30T06:25:53.134ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81192025-04-30T06:25:53.137ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81202025-04-30T06:25:53.144ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81212025-04-30T06:25:53.146ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81222025-04-30T06:25:53.146ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81232025-04-30T06:25:53.146ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81242025-04-30T06:25:53.189ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
81252025-04-30T06:25:53.191ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
81262025-04-30T06:25:53.192ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
81272025-04-30T06:25:53.192ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 689.547538ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
81282025-04-30T06:25:53.192ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
81292025-04-30T06:25:53.192ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
81302025-04-30T06:25:53.192ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81312025-04-30T06:25:53.192ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81322025-04-30T06:25:53.192ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81332025-04-30T06:25:53.192ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81342025-04-30T06:25:53.194ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81352025-04-30T06:25:53.198ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
81362025-04-30T06:25:53.198ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
81372025-04-30T06:25:53.198ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
81382025-04-30T06:25:53.198ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
81392025-04-30T06:25:53.216ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
81402025-04-30T06:25:53.216ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
81412025-04-30T06:25:53.220ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 52 03 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 R............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62130 req_id = d7fed94d-8c39-4ebd-9d2c-91b3758e1432
81422025-04-30T06:25:53.220ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20995,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62130 req_id = d7fed94d-8c39-4ebd-9d2c-91b3758e1432
81432025-04-30T06:25:53.221ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
81442025-04-30T06:25:53.221ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
81452025-04-30T06:25:53.221ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
81462025-04-30T06:25:53.221ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62130 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d7fed94d-8c39-4ebd-9d2c-91b3758e1432
81472025-04-30T06:25:53.222ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3218, tv_nsec: 21427361 } })
81482025-04-30T06:25:53.222ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
81492025-04-30T06:25:53.222ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
81502025-04-30T06:25:53.223ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0f 71 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .q..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53606 req_id = 9bfbae43-4d9e-4f2a-9eaf-d1a01de3e3c1
81512025-04-30T06:25:53.223ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3953,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53606 req_id = 9bfbae43-4d9e-4f2a-9eaf-d1a01de3e3c1
81522025-04-30T06:25:53.223ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
81532025-04-30T06:25:53.223ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53606 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9bfbae43-4d9e-4f2a-9eaf-d1a01de3e3c1
81542025-04-30T06:25:53.223ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 41 81 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e A...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61609 req_id = e3ee5b3e-cac3-4d94-95e6-3951c0b635d3
81552025-04-30T06:25:53.223ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16769,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61609 req_id = e3ee5b3e-cac3-4d94-95e6-3951c0b635d3
81562025-04-30T06:25:53.224ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
81572025-04-30T06:25:53.224ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61609 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e3ee5b3e-cac3-4d94-95e6-3951c0b635d3
81582025-04-30T06:25:53.387ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
81592025-04-30T06:25:53.388ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
81602025-04-30T06:25:53.388ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81612025-04-30T06:25:53.388ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81622025-04-30T06:25:53.388ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
81632025-04-30T06:25:53.388ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81642025-04-30T06:25:53.389ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
81652025-04-30T06:25:53.389ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81662025-04-30T06:25:53.390ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b67925c2-1bb9-45f6-a4e0-cf317c530f9e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81672025-04-30T06:25:53.390ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 12 42 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .B..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60444 req_id = 8796b9b6-47a3-4ea7-98a9-fa45bd87ac43
81682025-04-30T06:25:53.390ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4674,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60444 req_id = 8796b9b6-47a3-4ea7-98a9-fa45bd87ac43
81692025-04-30T06:25:53.390ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
81702025-04-30T06:25:53.390ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4674, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60444 req_id = 8796b9b6-47a3-4ea7-98a9-fa45bd87ac43
81712025-04-30T06:25:53.391ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b67925c2-1bb9-45f6-a4e0-cf317c530f9e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81722025-04-30T06:25:53.391ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b67925c2-1bb9-45f6-a4e0-cf317c530f9e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81732025-04-30T06:25:53.391ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b67925c2-1bb9-45f6-a4e0-cf317c530f9e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81742025-04-30T06:25:53.393ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
81752025-04-30T06:25:53.393ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
81762025-04-30T06:25:53.393ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
81772025-04-30T06:25:53.393ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81782025-04-30T06:25:53.398ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81792025-04-30T06:25:53.398ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81802025-04-30T06:25:53.398ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
81812025-04-30T06:25:53.399ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81822025-04-30T06:25:53.399ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 9506 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b67925c2-1bb9-45f6-a4e0-cf317c530f9e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
81832025-04-30T06:25:53.400ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b67925c2-1bb9-45f6-a4e0-cf317c530f9e", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
81842025-04-30T06:25:53.400ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
81852025-04-30T06:25:53.401ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = c25c06f2-0e8a-4f62-8bd2-43aae13f583e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
81862025-04-30T06:25:53.401ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = c25c06f2-0e8a-4f62-8bd2-43aae13f583e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
81872025-04-30T06:25:53.401ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = c25c06f2-0e8a-4f62-8bd2-43aae13f583e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
81882025-04-30T06:25:53.402ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = c25c06f2-0e8a-4f62-8bd2-43aae13f583e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
81892025-04-30T06:25:53.402ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6b469ee-6f69-4719-8a32-a08d861f1aa7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
81902025-04-30T06:25:53.402ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
81912025-04-30T06:25:53.402ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81922025-04-30T06:25:53.402ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
81932025-04-30T06:25:53.403ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81942025-04-30T06:25:53.405ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "73256c3e-d580-4081-84c7-ddc3d66b5b08", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
81952025-04-30T06:25:53.405ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
81962025-04-30T06:25:53.405ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 4594 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = c25c06f2-0e8a-4f62-8bd2-43aae13f583e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
81972025-04-30T06:25:53.406ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c25c06f2-0e8a-4f62-8bd2-43aae13f583e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
81982025-04-30T06:25:53.406ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81992025-04-30T06:25:53.406ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
82002025-04-30T06:25:53.406ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
82012025-04-30T06:25:53.408ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82022025-04-30T06:25:53.408ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82032025-04-30T06:25:53.409ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82042025-04-30T06:25:53.410ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82052025-04-30T06:25:53.410ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82062025-04-30T06:25:53.410ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
82072025-04-30T06:25:53.411ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee2557b4-247b-4b70-b894-6eb2c69c2f21", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82082025-04-30T06:25:53.411ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
82092025-04-30T06:25:53.414ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e107e0-c9a8-4030-a108-1aa5f9a0e37d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82102025-04-30T06:25:53.414ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
82112025-04-30T06:25:53.417ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccc28835-24f5-4fe2-bab7-0a39aba09b73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82122025-04-30T06:25:53.417ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
82132025-04-30T06:25:53.419ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff62c6d4-b2c2-40d2-ad7e-3ae4b6345fc0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82142025-04-30T06:25:53.419ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
82152025-04-30T06:25:53.421ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bddb49a5-1840-49c8-bab0-aecf65786f1c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82162025-04-30T06:25:53.421ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
82172025-04-30T06:25:53.425ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f71da8a-0883-440c-a034-4b6855e0c941", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82182025-04-30T06:25:53.426ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
82192025-04-30T06:25:53.433ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d3856d9-02f8-4ae7-b86c-0bc37a2f1152", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82202025-04-30T06:25:53.433ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
82212025-04-30T06:25:53.439ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
82222025-04-30T06:25:53.471ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c617a11-841b-4ad6-9473-93c9c2f6f6e6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82232025-04-30T06:25:53.471ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
82242025-04-30T06:25:53.485ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a30d933-5d45-4976-9635-82b304b4f83e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82252025-04-30T06:25:53.485ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
82262025-04-30T06:25:53.493ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82272025-04-30T06:25:53.493ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82282025-04-30T06:25:53.493ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82292025-04-30T06:25:53.493ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
82302025-04-30T06:25:53.494ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82312025-04-30T06:25:53.496ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8939b9cf-3bd5-4fed-a4e7-e6d4c30cb332", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82322025-04-30T06:25:53.497ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
82332025-04-30T06:25:53.498ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82342025-04-30T06:25:53.503ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82352025-04-30T06:25:53.504ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82362025-04-30T06:25:53.504ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
82372025-04-30T06:25:53.506ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82382025-04-30T06:25:53.520ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d1db8d1-012d-4928-b910-d3254e097ed2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82392025-04-30T06:25:53.520ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
82402025-04-30T06:25:53.520ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82412025-04-30T06:25:53.520ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
82422025-04-30T06:25:53.521ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82432025-04-30T06:25:53.526ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbaf0f07-f449-4abf-9ec3-8edd4842821c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82442025-04-30T06:25:53.526ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
82452025-04-30T06:25:53.531ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "985f9584-56df-497a-9ad3-fee0f596002f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82462025-04-30T06:25:53.531ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
82472025-04-30T06:25:53.535ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b95512f8-d832-44af-b61c-b7c5e5dd94d1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82482025-04-30T06:25:53.535ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
82492025-04-30T06:25:53.541ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "509f4784-ca39-4ec3-9c24-3b765956df09", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82502025-04-30T06:25:53.541ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
82512025-04-30T06:25:53.545ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b7dd39-ef2b-415d-8ea3-6b627b2f7b9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82522025-04-30T06:25:53.546ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
82532025-04-30T06:25:53.551ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
82542025-04-30T06:25:53.553ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98f28b7f-f541-4083-a5ab-cff114107d3c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82552025-04-30T06:25:53.554ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
82562025-04-30T06:25:53.555ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96ff4ee8-f1e4-4d3c-a49f-cc9712b5dd6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82572025-04-30T06:25:53.555ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
82582025-04-30T06:25:53.557ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a51cba0e-44f6-4bbd-9076-36e7f6666ca2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82592025-04-30T06:25:53.564ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
82602025-04-30T06:25:53.565ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc0487b5-3e8e-4437-9579-a88a77eedcce", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82612025-04-30T06:25:53.566ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
82622025-04-30T06:25:53.567ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "051bb2e2-6403-4502-b80c-a1710e59de47", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82632025-04-30T06:25:53.567ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
82642025-04-30T06:25:53.569ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1190b56d-1bbb-4b36-9ecf-cced854e2c97", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82652025-04-30T06:25:53.570ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
82662025-04-30T06:25:53.574ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e627b659-4377-4e96-a2c4-66e90cfe5fe5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82672025-04-30T06:25:53.574ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
82682025-04-30T06:25:53.579ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec552899-cf7b-4785-916b-a4720d26c389", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82692025-04-30T06:25:53.579ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
82702025-04-30T06:25:53.583ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a0b290-f0d3-4cc4-aa49-c79dedc219cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82712025-04-30T06:25:53.583ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
82722025-04-30T06:25:53.590ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a204ef6f-89f0-4bed-bc88-d60ca96e7677", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82732025-04-30T06:25:53.590ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
82742025-04-30T06:25:53.592ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc969e6d-cc76-4997-898f-0a3587c8476c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82752025-04-30T06:25:53.592ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
82762025-04-30T06:25:53.594ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6c0278a-4188-42c9-a4e8-97a2a9790fa5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82772025-04-30T06:25:53.595ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
82782025-04-30T06:25:53.596ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3ab6805-39bb-4182-a619-6ef5f591359e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82792025-04-30T06:25:53.596ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
82802025-04-30T06:25:53.597ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cca73bec-9890-4c3b-9b26-f5562417b00a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82812025-04-30T06:25:53.597ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
82822025-04-30T06:25:53.598ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c31ab673-5515-4260-bc1f-e177d4305f58", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82832025-04-30T06:25:53.598ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
82842025-04-30T06:25:53.599ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "995a3223-8580-4290-b432-1b73f99b9556", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82852025-04-30T06:25:53.601ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
82862025-04-30T06:25:53.602ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82872025-04-30T06:25:53.602ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82882025-04-30T06:25:53.602ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82892025-04-30T06:25:53.602ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
82902025-04-30T06:25:53.603ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82912025-04-30T06:25:53.605ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc454736-6e79-45a5-a4e1-1eb5328491b4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
82922025-04-30T06:25:53.605ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
82932025-04-30T06:25:53.607ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82942025-04-30T06:25:53.613ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82952025-04-30T06:25:53.613ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82962025-04-30T06:25:53.613ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
82972025-04-30T06:25:53.613ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82982025-04-30T06:25:53.614ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82992025-04-30T06:25:53.614ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
83002025-04-30T06:25:53.614ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
83012025-04-30T06:25:53.615ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
83022025-04-30T06:25:53.621ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
83032025-04-30T06:25:53.621ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
83042025-04-30T06:25:53.627ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
83052025-04-30T06:25:53.627ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
83062025-04-30T06:25:53.628ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83072025-04-30T06:25:53.628ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
83082025-04-30T06:25:53.629ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83092025-04-30T06:25:53.632ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "243f094c-c56c-4eb5-88de-b7357855d2e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83102025-04-30T06:25:53.632ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
83112025-04-30T06:25:53.639ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "478f11db-2cf7-47d3-9b7d-717edab2d684", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83122025-04-30T06:25:53.640ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
83132025-04-30T06:25:53.650ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e35458-f893-414a-91cb-d2565c124b28", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83142025-04-30T06:25:53.650ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
83152025-04-30T06:25:53.656ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88d22163-3287-4f08-8e64-ee9303a3bba4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83162025-04-30T06:25:53.656ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
83172025-04-30T06:25:53.660ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7f53cfa-5863-40a5-808c-e04350a3770f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83182025-04-30T06:25:53.661ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
83192025-04-30T06:25:53.666ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
83202025-04-30T06:25:53.668ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9137f490-1e1a-4bee-9790-2bccdb228a99", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83212025-04-30T06:25:53.668ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
83222025-04-30T06:25:53.670ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2be7b0e9-63cb-46ed-9987-cbd5c7718416", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83232025-04-30T06:25:53.671ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
83242025-04-30T06:25:53.679ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0748bc0f-1a52-436c-9913-793d731c74fc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83252025-04-30T06:25:53.679ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
83262025-04-30T06:25:53.683ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f6555f-05c2-4502-857f-48eb5fbd0438", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83272025-04-30T06:25:53.684ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
83282025-04-30T06:25:53.687ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57ebdeee-ccf2-4e80-9d5b-cf86b0772e0a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83292025-04-30T06:25:53.687ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
83302025-04-30T06:25:53.688ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
83312025-04-30T06:25:53.690ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "60afe089-7ec7-4d86-acb8-9090ca3a526e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83322025-04-30T06:25:53.691ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
83332025-04-30T06:25:53.692ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d80d6f8-cacb-412a-a043-a8bdf4799e3f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83342025-04-30T06:25:53.693ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
83352025-04-30T06:25:53.695ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97b79a00-5094-4578-ad77-d22eb2211742", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83362025-04-30T06:25:53.695ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
83372025-04-30T06:25:53.697ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b425c015-28bf-455a-b98b-6b4a3965e65e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83382025-04-30T06:25:53.697ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
83392025-04-30T06:25:53.698ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9384b26-80a4-4d25-9b5f-114722f05535", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83402025-04-30T06:25:53.699ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83412025-04-30T06:25:53.699ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83422025-04-30T06:25:53.699ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83432025-04-30T06:25:53.699ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83442025-04-30T06:25:53.700ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83452025-04-30T06:25:53.704ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83462025-04-30T06:25:53.708ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83472025-04-30T06:25:53.709ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
83482025-04-30T06:25:53.709ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83492025-04-30T06:25:53.709ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83502025-04-30T06:25:53.709ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83512025-04-30T06:25:53.709ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83522025-04-30T06:25:53.712ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83532025-04-30T06:25:53.719ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
83542025-04-30T06:25:53.719ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83552025-04-30T06:25:53.719ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83562025-04-30T06:25:53.720ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83572025-04-30T06:25:53.720ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83582025-04-30T06:25:53.720ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83592025-04-30T06:25:53.720ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83602025-04-30T06:25:53.720ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
83612025-04-30T06:25:53.721ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83622025-04-30T06:25:53.721ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
83632025-04-30T06:25:53.724ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83642025-04-30T06:25:53.737ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83652025-04-30T06:25:53.737ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83662025-04-30T06:25:53.737ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
83672025-04-30T06:25:53.738ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83682025-04-30T06:25:53.795ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
83692025-04-30T06:25:53.795ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
83702025-04-30T06:25:53.796ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:64971 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
83712025-04-30T06:25:53.797ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:64971 req_id = ca9ed9bd-e874-4da8-888d-3e68586cf417 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
83722025-04-30T06:25:53.797ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 410 local_addr = [::1]:42668 method = GET remote_addr = [::1]:64971 req_id = ca9ed9bd-e874-4da8-888d-3e68586cf417 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
83732025-04-30T06:25:53.797ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca9ed9bd-e874-4da8-888d-3e68586cf417", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83742025-04-30T06:25:53.880ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
83752025-04-30T06:25:53.880ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
83762025-04-30T06:25:53.880ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83772025-04-30T06:25:53.881ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
83782025-04-30T06:25:53.881ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83792025-04-30T06:25:53.882ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:40497 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
83802025-04-30T06:25:53.883ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:40497 req_id = f47ad259-3e80-4404-9c58-0c5b1e4eb36f sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
83812025-04-30T06:25:53.884ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 335 local_addr = [::1]:47918 method = GET remote_addr = [::1]:40497 req_id = f47ad259-3e80-4404-9c58-0c5b1e4eb36f response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
83822025-04-30T06:25:53.884ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f47ad259-3e80-4404-9c58-0c5b1e4eb36f", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83832025-04-30T06:25:53.942ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
83842025-04-30T06:25:53.942ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
83852025-04-30T06:25:53.947ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:64032 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
83862025-04-30T06:25:53.948ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:64032 req_id = 7acc7c34-13c2-430b-b35d-ddc13f54bf3d sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
83872025-04-30T06:25:53.949ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 413 local_addr = [::1]:35744 method = GET remote_addr = [::1]:64032 req_id = 7acc7c34-13c2-430b-b35d-ddc13f54bf3d response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
83882025-04-30T06:25:53.949ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7acc7c34-13c2-430b-b35d-ddc13f54bf3d", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:52 GMT"} })
83892025-04-30T06:25:53.998ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
83902025-04-30T06:25:53.998ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
83912025-04-30T06:25:53.998ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:39916 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83922025-04-30T06:25:54.001ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:39916 req_id = 99860134-4dcf-4089-93c4-c6b9e1722bfe sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83932025-04-30T06:25:54.002ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1228 local_addr = [::1]:35343 method = GET remote_addr = [::1]:39916 req_id = 99860134-4dcf-4089-93c4-c6b9e1722bfe response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83942025-04-30T06:25:54.002ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "99860134-4dcf-4089-93c4-c6b9e1722bfe", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
83952025-04-30T06:25:54.004ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83962025-04-30T06:25:54.004ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83972025-04-30T06:25:54.004ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
83982025-04-30T06:25:54.009ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83992025-04-30T06:25:54.009ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84002025-04-30T06:25:54.009ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84012025-04-30T06:25:54.009ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84022025-04-30T06:25:54.010ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84032025-04-30T06:25:54.013ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84042025-04-30T06:25:54.020ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84052025-04-30T06:25:54.020ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84062025-04-30T06:25:54.020ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84072025-04-30T06:25:54.021ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84082025-04-30T06:25:54.045ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
84092025-04-30T06:25:54.097ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84102025-04-30T06:25:54.098ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84112025-04-30T06:25:54.098ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84122025-04-30T06:25:54.098ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
84132025-04-30T06:25:54.103ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84142025-04-30T06:25:54.111ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84152025-04-30T06:25:54.118ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84162025-04-30T06:25:54.118ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84172025-04-30T06:25:54.118ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
84182025-04-30T06:25:54.119ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84192025-04-30T06:25:54.124ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
84202025-04-30T06:25:54.124ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
84212025-04-30T06:25:54.124ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 931.118362ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
84222025-04-30T06:25:54.124ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
84232025-04-30T06:25:54.124ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
84242025-04-30T06:25:54.124ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84252025-04-30T06:25:54.124ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84262025-04-30T06:25:54.124ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84272025-04-30T06:25:54.124ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84282025-04-30T06:25:54.130ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84292025-04-30T06:25:54.130ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
84302025-04-30T06:25:54.131ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84312025-04-30T06:25:54.131ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84322025-04-30T06:25:54.141ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
84332025-04-30T06:25:54.141ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84342025-04-30T06:25:54.141ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84352025-04-30T06:25:54.142ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84362025-04-30T06:25:54.162ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
84372025-04-30T06:25:54.162ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
84382025-04-30T06:25:54.175ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ba e9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57131 req_id = 90151124-3602-4da0-9d12-243d72235412
84392025-04-30T06:25:54.176ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47849,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57131 req_id = 90151124-3602-4da0-9d12-243d72235412
84402025-04-30T06:25:54.176ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
84412025-04-30T06:25:54.177ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
84422025-04-30T06:25:54.178ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84432025-04-30T06:25:54.178ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57131 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 90151124-3602-4da0-9d12-243d72235412
84442025-04-30T06:25:54.181ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3218, tv_nsec: 980863455 } })
84452025-04-30T06:25:54.182ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
84462025-04-30T06:25:54.182ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
84472025-04-30T06:25:54.191ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 06 29 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .)..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51872 req_id = 1f15a9c5-fc5f-4999-b724-ff6441052b8f
84482025-04-30T06:25:54.191ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1577,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51872 req_id = 1f15a9c5-fc5f-4999-b724-ff6441052b8f
84492025-04-30T06:25:54.191ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
84502025-04-30T06:25:54.192ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51872 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1f15a9c5-fc5f-4999-b724-ff6441052b8f
84512025-04-30T06:25:54.192ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5d 65 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ]e..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64266 req_id = 22d59cdc-a66d-4fce-887c-cbcf4a10741b
84522025-04-30T06:25:54.192ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23909,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64266 req_id = 22d59cdc-a66d-4fce-887c-cbcf4a10741b
84532025-04-30T06:25:54.192ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
84542025-04-30T06:25:54.195ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64266 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 22d59cdc-a66d-4fce-887c-cbcf4a10741b
84552025-04-30T06:25:54.407ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
84562025-04-30T06:25:54.411ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84572025-04-30T06:25:54.412ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84582025-04-30T06:25:54.428ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 91 a8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55009 req_id = 1a171017-8956-43cb-8325-27ec0dde4005
84592025-04-30T06:25:54.428ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37288,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55009 req_id = 1a171017-8956-43cb-8325-27ec0dde4005
84602025-04-30T06:25:54.428ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
84612025-04-30T06:25:54.428ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37288, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55009 req_id = 1a171017-8956-43cb-8325-27ec0dde4005
84622025-04-30T06:25:54.434ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
84632025-04-30T06:25:54.434ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
84642025-04-30T06:25:54.434ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
84652025-04-30T06:25:54.436ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84662025-04-30T06:25:54.441ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84672025-04-30T06:25:54.441ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84682025-04-30T06:25:54.441ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
84692025-04-30T06:25:54.479ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "10f64cda-f6ee-4a79-8f28-9143e6651966", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84702025-04-30T06:25:54.480ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
84712025-04-30T06:25:54.480ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
84722025-04-30T06:25:54.496ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77400b92-bc8d-4189-93ce-c988370034fa", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84732025-04-30T06:25:54.497ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
84742025-04-30T06:25:54.510ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4250d9d7-b400-4629-8eea-66369b35ae92", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84752025-04-30T06:25:54.510ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
84762025-04-30T06:25:54.512ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8520e1b-f9f6-443c-9eea-74f9cedd4c3c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84772025-04-30T06:25:54.513ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
84782025-04-30T06:25:54.515ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4188a666-b56f-458b-9220-3bc638c25002", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84792025-04-30T06:25:54.515ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
84802025-04-30T06:25:54.517ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51838d9a-b40c-43ac-9da6-90e4dd94fa0e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84812025-04-30T06:25:54.517ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
84822025-04-30T06:25:54.522ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0e81665-ff4f-428e-b1b2-60d2f422883c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84832025-04-30T06:25:54.522ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
84842025-04-30T06:25:54.526ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4b22dc5-f08b-4de5-871e-ae589871ed00", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84852025-04-30T06:25:54.526ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
84862025-04-30T06:25:54.527ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27e63738-4efa-46ef-849f-86b211d0d5a7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84872025-04-30T06:25:54.527ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
84882025-04-30T06:25:54.528ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9671de09-458d-4075-9d93-1c30c4ff05ec", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84892025-04-30T06:25:54.528ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
84902025-04-30T06:25:54.529ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a75d2357-2f6a-4c73-995b-7a7ca1a0c933", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84912025-04-30T06:25:54.529ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
84922025-04-30T06:25:54.531ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fe9a8a1-72a3-4e92-a1cf-3d781f496f5a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
84932025-04-30T06:25:54.531ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
84942025-04-30T06:25:54.531ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84952025-04-30T06:25:54.531ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84962025-04-30T06:25:54.531ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84972025-04-30T06:25:54.531ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
84982025-04-30T06:25:54.533ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84992025-04-30T06:25:54.536ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aeb75ef7-0954-4f6f-8d6b-607e7e1a17df", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85002025-04-30T06:25:54.536ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
85012025-04-30T06:25:54.537ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85022025-04-30T06:25:54.542ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85032025-04-30T06:25:54.542ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85042025-04-30T06:25:54.542ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
85052025-04-30T06:25:54.543ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85062025-04-30T06:25:54.546ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85072025-04-30T06:25:54.546ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
85082025-04-30T06:25:54.546ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85092025-04-30T06:25:54.547ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4d8477f-baf7-454c-bf36-cbe12e75fa5e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85102025-04-30T06:25:54.547ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
85112025-04-30T06:25:54.549ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a689ebe2-be1e-4342-8771-7034ec3674a6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85122025-04-30T06:25:54.550ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
85132025-04-30T06:25:54.552ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff6b8ce3-aa83-45d8-a362-ef4af590e0c0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85142025-04-30T06:25:54.553ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
85152025-04-30T06:25:54.556ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df88fb24-0c3b-4670-bd04-7dc228db8576", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85162025-04-30T06:25:54.557ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
85172025-04-30T06:25:54.560ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fafc004-3695-4e13-91a8-3a48c5eaa428", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85182025-04-30T06:25:54.560ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
85192025-04-30T06:25:54.562ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed3298ad-0370-4dab-b153-21a04305e814", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85202025-04-30T06:25:54.562ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
85212025-04-30T06:25:54.564ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f3ea1c1-1d70-4100-87c9-53f0e8c98021", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85222025-04-30T06:25:54.564ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
85232025-04-30T06:25:54.566ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "534f9416-3c5f-4683-9b2f-9b58dc2f0044", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85242025-04-30T06:25:54.567ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
85252025-04-30T06:25:54.568ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ca8635f-86a2-4992-a3c6-9c4b0d653796", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85262025-04-30T06:25:54.568ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
85272025-04-30T06:25:54.572ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d513e6-aae7-49fc-8304-bb113e19cc9b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85282025-04-30T06:25:54.572ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
85292025-04-30T06:25:54.577ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
85302025-04-30T06:25:54.577ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f821affb-e4db-47a8-8244-e394c5ef6cce", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85312025-04-30T06:25:54.577ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
85322025-04-30T06:25:54.580ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "61fffb9a-d23d-4d50-a394-1a22a2c8d13e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85332025-04-30T06:25:54.580ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
85342025-04-30T06:25:54.582ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b24aa4fd-5a28-4feb-a0ad-ab871d5ff953", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85352025-04-30T06:25:54.582ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
85362025-04-30T06:25:54.585ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d560b3e-c5c0-4f9b-931c-2a2f968e66d5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85372025-04-30T06:25:54.586ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
85382025-04-30T06:25:54.589ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5d693b4-fdfd-4647-ad7f-73d5344ced60", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85392025-04-30T06:25:54.590ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
85402025-04-30T06:25:54.596ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ae7ca9c-4be8-4611-bbfb-5d9fbdb094d6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85412025-04-30T06:25:54.597ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
85422025-04-30T06:25:54.605ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5b17193-ae4d-4d3f-8e8e-fe804070025d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85432025-04-30T06:25:54.605ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
85442025-04-30T06:25:54.607ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85452025-04-30T06:25:54.607ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85462025-04-30T06:25:54.608ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
85472025-04-30T06:25:54.612ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
85482025-04-30T06:25:54.619ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
85492025-04-30T06:25:54.620ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
85502025-04-30T06:25:54.621ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9074f2b-703f-4c0e-a3fa-47af60e7a804", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85512025-04-30T06:25:54.621ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
85522025-04-30T06:25:54.626ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
85532025-04-30T06:25:54.626ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85542025-04-30T06:25:54.627ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e303c191-28cb-4b70-8798-7064ccf45769", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85552025-04-30T06:25:54.627ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
85562025-04-30T06:25:54.628ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85572025-04-30T06:25:54.628ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85582025-04-30T06:25:54.629ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85592025-04-30T06:25:54.629ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
85602025-04-30T06:25:54.630ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85612025-04-30T06:25:54.632ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6366b5e7-7471-406d-809a-11b90c656079", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85622025-04-30T06:25:54.633ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
85632025-04-30T06:25:54.634ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85642025-04-30T06:25:54.642ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85652025-04-30T06:25:54.643ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85662025-04-30T06:25:54.643ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
85672025-04-30T06:25:54.644ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85682025-04-30T06:25:54.646ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "736210f2-c8fb-4c99-812c-8fbab21ce542", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85692025-04-30T06:25:54.646ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
85702025-04-30T06:25:54.648ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85712025-04-30T06:25:54.648ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
85722025-04-30T06:25:54.649ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85732025-04-30T06:25:54.650ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "139cb98a-5ad9-4f38-a52a-5ee9fa388701", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85742025-04-30T06:25:54.650ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
85752025-04-30T06:25:54.654ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b4ba043-546f-45ce-9264-26a72b94959f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85762025-04-30T06:25:54.654ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
85772025-04-30T06:25:54.659ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe58b643-5c2a-4779-80b6-8975a7101a51", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85782025-04-30T06:25:54.659ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
85792025-04-30T06:25:54.664ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e666eb0-c1f5-474c-b9cf-10204039aa00", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85802025-04-30T06:25:54.664ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
85812025-04-30T06:25:54.671ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6346fac-1c7f-4a4f-8680-1acb4bdff064", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85822025-04-30T06:25:54.671ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
85832025-04-30T06:25:54.677ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c70cac9-3878-4e3a-82bd-a4e60cf49756", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85842025-04-30T06:25:54.677ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
85852025-04-30T06:25:54.682ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
85862025-04-30T06:25:54.684ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbccc1d8-f87f-419e-b32d-7b9b04299d39", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85872025-04-30T06:25:54.684ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
85882025-04-30T06:25:54.686ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7043e07-495a-403a-bac1-65ed6a019d33", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85892025-04-30T06:25:54.686ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
85902025-04-30T06:25:54.687ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc7804aa-897a-4616-9bec-0773c9787dd8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85912025-04-30T06:25:54.687ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
85922025-04-30T06:25:54.690ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c575c15b-8b14-4090-8446-ad7e3dbada47", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85932025-04-30T06:25:54.690ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
85942025-04-30T06:25:54.691ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce66c427-404c-43fc-a964-259a36175205", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85952025-04-30T06:25:54.691ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
85962025-04-30T06:25:54.691ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
85972025-04-30T06:25:54.695ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c583ed5f-4acf-4764-98d4-8edf55e0ddc2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
85982025-04-30T06:25:54.695ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
85992025-04-30T06:25:54.706ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8298afd3-baa6-4fed-a346-40a36c3270a4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
86002025-04-30T06:25:54.706ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
86012025-04-30T06:25:54.714ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1eb5cb93-7d25-40db-9b11-e303a3d5f18b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
86022025-04-30T06:25:54.714ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
86032025-04-30T06:25:54.714ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
86042025-04-30T06:25:54.715ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f07eaac-363c-4fd9-9e4b-6b6a274e61ae", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
86052025-04-30T06:25:54.715ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
86062025-04-30T06:25:54.718ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91dd182a-76bf-4dd5-b060-5c7dd2d7fe17", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
86072025-04-30T06:25:54.718ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86082025-04-30T06:25:54.719ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86092025-04-30T06:25:54.719ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86102025-04-30T06:25:54.719ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86112025-04-30T06:25:54.719ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86122025-04-30T06:25:54.722ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86132025-04-30T06:25:54.730ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86142025-04-30T06:25:54.730ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86152025-04-30T06:25:54.730ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86162025-04-30T06:25:54.730ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86172025-04-30T06:25:54.730ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86182025-04-30T06:25:54.731ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86192025-04-30T06:25:54.733ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86202025-04-30T06:25:54.733ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86212025-04-30T06:25:54.733ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86222025-04-30T06:25:54.733ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
86232025-04-30T06:25:54.733ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86242025-04-30T06:25:54.734ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86252025-04-30T06:25:54.739ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
86262025-04-30T06:25:54.740ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86272025-04-30T06:25:54.740ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86282025-04-30T06:25:54.740ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86292025-04-30T06:25:54.742ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86302025-04-30T06:25:54.749ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86312025-04-30T06:25:54.749ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86322025-04-30T06:25:54.749ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
86332025-04-30T06:25:54.750ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86342025-04-30T06:25:54.833ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
86352025-04-30T06:25:54.833ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
86362025-04-30T06:25:54.841ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:45575 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
86372025-04-30T06:25:54.842ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:45575 req_id = f4292d50-99a9-449f-a98b-1dc8c9a3abfb sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
86382025-04-30T06:25:54.843ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 721 local_addr = [::1]:42668 method = GET remote_addr = [::1]:45575 req_id = f4292d50-99a9-449f-a98b-1dc8c9a3abfb response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
86392025-04-30T06:25:54.844ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86402025-04-30T06:25:54.844ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
86412025-04-30T06:25:54.846ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86422025-04-30T06:25:54.846ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4292d50-99a9-449f-a98b-1dc8c9a3abfb", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
86432025-04-30T06:25:54.907ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
86442025-04-30T06:25:54.907ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
86452025-04-30T06:25:54.907ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:51055 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
86462025-04-30T06:25:54.911ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:51055 req_id = 81b0f23e-6243-405c-8b09-134202188719 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
86472025-04-30T06:25:54.912ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 387 local_addr = [::1]:47918 method = GET remote_addr = [::1]:51055 req_id = 81b0f23e-6243-405c-8b09-134202188719 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
86482025-04-30T06:25:54.918ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "81b0f23e-6243-405c-8b09-134202188719", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:54 GMT"} })
86492025-04-30T06:25:55.024ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
86502025-04-30T06:25:55.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
86512025-04-30T06:25:55.027ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:35566 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
86522025-04-30T06:25:55.029ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:35566 req_id = 62b4668f-7e3f-4ecb-b7ff-b1b92022f069 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
86532025-04-30T06:25:55.032ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3255 local_addr = [::1]:35744 method = GET remote_addr = [::1]:35566 req_id = 62b4668f-7e3f-4ecb-b7ff-b1b92022f069 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
86542025-04-30T06:25:55.036ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "62b4668f-7e3f-4ecb-b7ff-b1b92022f069", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:55 GMT"} })
86552025-04-30T06:25:55.103ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
86562025-04-30T06:25:55.104ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
86572025-04-30T06:25:55.108ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:42299 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
86582025-04-30T06:25:55.109ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:42299 req_id = c689510a-098b-486c-9461-e184640bc2a9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86592025-04-30T06:25:55.110ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1489 local_addr = [::1]:35343 method = GET remote_addr = [::1]:42299 req_id = c689510a-098b-486c-9461-e184640bc2a9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
86602025-04-30T06:25:55.111ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c689510a-098b-486c-9461-e184640bc2a9", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:55 GMT"} })
86612025-04-30T06:25:55.117ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86622025-04-30T06:25:55.117ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
86632025-04-30T06:25:55.117ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
86642025-04-30T06:25:55.123ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86652025-04-30T06:25:55.123ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86662025-04-30T06:25:55.123ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86672025-04-30T06:25:55.123ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86682025-04-30T06:25:55.124ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86692025-04-30T06:25:55.136ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86702025-04-30T06:25:55.148ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86712025-04-30T06:25:55.149ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86722025-04-30T06:25:55.149ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86732025-04-30T06:25:55.149ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86742025-04-30T06:25:55.187ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
86752025-04-30T06:25:55.240ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86762025-04-30T06:25:55.241ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86772025-04-30T06:25:55.241ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86782025-04-30T06:25:55.241ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
86792025-04-30T06:25:55.242ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86802025-04-30T06:25:55.243ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
86812025-04-30T06:25:55.243ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
86822025-04-30T06:25:55.243ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.120508069s iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
86832025-04-30T06:25:55.244ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
86842025-04-30T06:25:55.244ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
86852025-04-30T06:25:55.244ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86862025-04-30T06:25:55.244ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86872025-04-30T06:25:55.244ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86882025-04-30T06:25:55.247ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86892025-04-30T06:25:55.251ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86902025-04-30T06:25:55.257ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
86912025-04-30T06:25:55.258ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
86922025-04-30T06:25:55.259ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
86932025-04-30T06:25:55.261ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
86942025-04-30T06:25:55.262ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86952025-04-30T06:25:55.285ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86962025-04-30T06:25:55.285ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86972025-04-30T06:25:55.285ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
86982025-04-30T06:25:55.285ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86992025-04-30T06:25:55.288ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
87002025-04-30T06:25:55.288ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
87012025-04-30T06:25:55.289ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 87 89 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63002 req_id = 02b4b9b7-bc2a-49a6-a6ae-cb944a09baa0
87022025-04-30T06:25:55.289ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34697,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63002 req_id = 02b4b9b7-bc2a-49a6-a6ae-cb944a09baa0
87032025-04-30T06:25:55.289ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
87042025-04-30T06:25:55.289ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
87052025-04-30T06:25:55.289ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87062025-04-30T06:25:55.289ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63002 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 02b4b9b7-bc2a-49a6-a6ae-cb944a09baa0
87072025-04-30T06:25:55.290ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3220, tv_nsec: 89204011 } })
87082025-04-30T06:25:55.290ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
87092025-04-30T06:25:55.290ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
87102025-04-30T06:25:55.290ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87112025-04-30T06:25:55.290ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
87122025-04-30T06:25:55.291ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87132025-04-30T06:25:55.291ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 11 26 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .&..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53544 req_id = 354723d5-c047-4f95-b1b3-6b28cd049e3b
87142025-04-30T06:25:55.291ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4390,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53544 req_id = 354723d5-c047-4f95-b1b3-6b28cd049e3b
87152025-04-30T06:25:55.291ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
87162025-04-30T06:25:55.291ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53544 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 354723d5-c047-4f95-b1b3-6b28cd049e3b
87172025-04-30T06:25:55.291ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e3 79 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .y..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62517 req_id = 79eb17a1-d7eb-478b-b766-ff2fc840fbf2
87182025-04-30T06:25:55.292ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58233,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62517 req_id = 79eb17a1-d7eb-478b-b766-ff2fc840fbf2
87192025-04-30T06:25:55.292ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87202025-04-30T06:25:55.292ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62517 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 79eb17a1-d7eb-478b-b766-ff2fc840fbf2
87212025-04-30T06:25:55.429ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
87222025-04-30T06:25:55.430ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
87232025-04-30T06:25:55.431ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
87242025-04-30T06:25:55.432ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
87252025-04-30T06:25:55.435ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
87262025-04-30T06:25:55.435ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
87272025-04-30T06:25:55.435ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
87282025-04-30T06:25:55.435ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
87292025-04-30T06:25:55.436ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
87302025-04-30T06:25:55.437ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
87312025-04-30T06:25:55.437ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
87322025-04-30T06:25:55.437ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
87332025-04-30T06:25:55.437ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 54.649\u{b5}s iteration = 3 status = {"error":"task disabled"}
87342025-04-30T06:25:55.437ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
87352025-04-30T06:25:55.437ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
87362025-04-30T06:25:55.437ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87372025-04-30T06:25:55.437ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
87382025-04-30T06:25:55.437ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87392025-04-30T06:25:55.438ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87402025-04-30T06:25:55.438ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
87412025-04-30T06:25:55.438ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
87422025-04-30T06:25:55.438ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87432025-04-30T06:25:55.438ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
87442025-04-30T06:25:55.438ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
87452025-04-30T06:25:55.438ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
87462025-04-30T06:25:55.438ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
87472025-04-30T06:25:55.439ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
87482025-04-30T06:25:55.439ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
87492025-04-30T06:25:55.440ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
87502025-04-30T06:25:55.441ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
87512025-04-30T06:25:55.441ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
87522025-04-30T06:25:55.441ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87532025-04-30T06:25:55.441ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
87542025-04-30T06:25:55.441ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
87552025-04-30T06:25:55.442ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
87562025-04-30T06:25:55.442ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
87572025-04-30T06:25:55.442ZINFOtest_replacement_sanity (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
87582025-04-30T06:25:55.442ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = instance_updater elapsed = 223.184\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
87592025-04-30T06:25:55.443ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
87602025-04-30T06:25:55.443ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
87612025-04-30T06:25:55.443ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
87622025-04-30T06:25:55.443ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87632025-04-30T06:25:55.443ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
87642025-04-30T06:25:55.443ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
87652025-04-30T06:25:55.443ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
87662025-04-30T06:25:55.443ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87672025-04-30T06:25:55.444ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87682025-04-30T06:25:55.445ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
87692025-04-30T06:25:55.447ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6a14d09a-29be-40b4-a772-1f6d632eaf24 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
87702025-04-30T06:25:55.447ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 94 60 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .`..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61995 req_id = 7b417c6b-ff2a-493f-9589-0a0e830f1846
87712025-04-30T06:25:55.447ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37984,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61995 req_id = 7b417c6b-ff2a-493f-9589-0a0e830f1846
87722025-04-30T06:25:55.448ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
87732025-04-30T06:25:55.448ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 37984, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61995 req_id = 7b417c6b-ff2a-493f-9589-0a0e830f1846
87742025-04-30T06:25:55.448ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6a14d09a-29be-40b4-a772-1f6d632eaf24 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
87752025-04-30T06:25:55.448ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6a14d09a-29be-40b4-a772-1f6d632eaf24 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
87762025-04-30T06:25:55.449ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6a14d09a-29be-40b4-a772-1f6d632eaf24 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
87772025-04-30T06:25:55.453ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
87782025-04-30T06:25:55.460ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
87792025-04-30T06:25:55.460ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
87802025-04-30T06:25:55.460ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
87812025-04-30T06:25:55.461ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87822025-04-30T06:25:55.462ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87832025-04-30T06:25:55.463ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87842025-04-30T06:25:55.463ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
87852025-04-30T06:25:55.463ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
87862025-04-30T06:25:55.470ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 33.860305ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
87872025-04-30T06:25:55.472ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87882025-04-30T06:25:55.487ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
87892025-04-30T06:25:55.487ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
87902025-04-30T06:25:55.487ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
87912025-04-30T06:25:55.488ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
87922025-04-30T06:25:55.493ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7b d4 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 {............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64340 req_id = 69c45a05-4fd1-4138-8a8c-7ad96d09833b
87932025-04-30T06:25:55.493ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31700,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64340 req_id = 69c45a05-4fd1-4138-8a8c-7ad96d09833b
87942025-04-30T06:25:55.493ZDEBGtest_replacement_sanity (store): query key key = _dendrite._tcp
87952025-04-30T06:25:55.494ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
87962025-04-30T06:25:55.494ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87972025-04-30T06:25:55.494ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64340 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 69c45a05-4fd1-4138-8a8c-7ad96d09833b
87982025-04-30T06:25:55.494ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87992025-04-30T06:25:55.503ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
88002025-04-30T06:25:55.503ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
88012025-04-30T06:25:55.503ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
88022025-04-30T06:25:55.505ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
88032025-04-30T06:25:55.509ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3220, tv_nsec: 308719151 } })
88042025-04-30T06:25:55.514ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
88052025-04-30T06:25:55.514ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
88062025-04-30T06:25:55.522ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: cc 91 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50544 req_id = 7d625924-8e88-46b0-a7ca-f6161dadb7cf
88072025-04-30T06:25:55.522ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52369,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50544 req_id = 7d625924-8e88-46b0-a7ca-f6161dadb7cf
88082025-04-30T06:25:55.523ZDEBGtest_replacement_sanity (store): query key key = _dendrite._tcp
88092025-04-30T06:25:55.523ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
88102025-04-30T06:25:55.524ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88112025-04-30T06:25:55.524ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50544 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 7d625924-8e88-46b0-a7ca-f6161dadb7cf
88122025-04-30T06:25:55.525ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88132025-04-30T06:25:55.541ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
88142025-04-30T06:25:55.541ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
88152025-04-30T06:25:55.541ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
88162025-04-30T06:25:55.545ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
88172025-04-30T06:25:55.547ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88182025-04-30T06:25:55.552ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88192025-04-30T06:25:55.553ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
88202025-04-30T06:25:55.553ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88212025-04-30T06:25:55.553ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
88222025-04-30T06:25:55.553ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88232025-04-30T06:25:55.553ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88242025-04-30T06:25:55.555ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3220, tv_nsec: 353877606 } })
88252025-04-30T06:25:55.555ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
88262025-04-30T06:25:55.555ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
88272025-04-30T06:25:55.559ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 112343 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6a14d09a-29be-40b4-a772-1f6d632eaf24 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
88282025-04-30T06:25:55.560ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
88292025-04-30T06:25:55.560ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
88302025-04-30T06:25:55.560ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
88312025-04-30T06:25:55.561ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
88322025-04-30T06:25:55.570ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 90 14 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54507 req_id = 63bf1953-5401-4e02-81bb-f8734c872c78
88332025-04-30T06:25:55.570ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36884,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54507 req_id = 63bf1953-5401-4e02-81bb-f8734c872c78
88342025-04-30T06:25:55.570ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
88352025-04-30T06:25:55.570ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54507 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 63bf1953-5401-4e02-81bb-f8734c872c78
88362025-04-30T06:25:55.570ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 58 88 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e X...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54254 req_id = 353b4643-336b-465f-a773-b2b4d1d8b715
88372025-04-30T06:25:55.570ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22664,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54254 req_id = 353b4643-336b-465f-a773-b2b4d1d8b715
88382025-04-30T06:25:55.570ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88392025-04-30T06:25:55.570ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54254 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 353b4643-336b-465f-a773-b2b4d1d8b715
88402025-04-30T06:25:55.571ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
88412025-04-30T06:25:55.571ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88422025-04-30T06:25:55.571ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88432025-04-30T06:25:55.571ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a14d09a-29be-40b4-a772-1f6d632eaf24", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:55 GMT"} })
88442025-04-30T06:25:55.571ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
88452025-04-30T06:25:55.574ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2526fe3e-fc0f-4c28-aba8-fd6b1fc746b8 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
88462025-04-30T06:25:55.574ZINFOtest_replacement_sanity (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
88472025-04-30T06:25:55.678ZINFOtest_replacement_sanity (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
88482025-04-30T06:25:55.678ZDEBGtest_replacement_sanity (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
88492025-04-30T06:25:55.679ZDEBGtest_replacement_sanity (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
88502025-04-30T06:25:55.679ZWARNtest_replacement_sanity (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
88512025-04-30T06:25:55.682ZWARNtest_replacement_sanity (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
88522025-04-30T06:25:55.683ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d3 30 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .0..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50871 req_id = 698b72f1-0a90-45b8-9738-0f328604b66a
88532025-04-30T06:25:55.683ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54064,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50871 req_id = 698b72f1-0a90-45b8-9738-0f328604b66a
88542025-04-30T06:25:55.683ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
88552025-04-30T06:25:55.683ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50871 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 698b72f1-0a90-45b8-9738-0f328604b66a
88562025-04-30T06:25:55.684ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1e 58 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .X..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58218 req_id = 92bba730-21a3-4d08-91cb-3ca9be44ac66
88572025-04-30T06:25:55.684ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7768,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58218 req_id = 92bba730-21a3-4d08-91cb-3ca9be44ac66
88582025-04-30T06:25:55.685ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88592025-04-30T06:25:55.687ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58218 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 92bba730-21a3-4d08-91cb-3ca9be44ac66
88602025-04-30T06:25:55.687ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 2c 12 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ,............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53780 req_id = c7c90ea1-b661-4d1b-a802-256ed6a60525
88612025-04-30T06:25:55.687ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11282,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53780 req_id = c7c90ea1-b661-4d1b-a802-256ed6a60525
88622025-04-30T06:25:55.687ZDEBGtest_replacement_sanity (store): query key key = _dendrite._tcp
88632025-04-30T06:25:55.687ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
88642025-04-30T06:25:55.688ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
88652025-04-30T06:25:55.688ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53780 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c7c90ea1-b661-4d1b-a802-256ed6a60525
88662025-04-30T06:25:55.688ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2526fe3e-fc0f-4c28-aba8-fd6b1fc746b8 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
88672025-04-30T06:25:55.688ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2526fe3e-fc0f-4c28-aba8-fd6b1fc746b8 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
88682025-04-30T06:25:55.690ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2526fe3e-fc0f-4c28-aba8-fd6b1fc746b8 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
88692025-04-30T06:25:55.690ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88702025-04-30T06:25:55.695ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
88712025-04-30T06:25:55.696ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
88722025-04-30T06:25:55.696ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
88732025-04-30T06:25:55.697ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
88742025-04-30T06:25:55.698ZINFOtest_replacement_sanity (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
88752025-04-30T06:25:55.698ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 257.213642ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
88762025-04-30T06:25:55.701ZINFOtest_replacement_sanity (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
88772025-04-30T06:25:55.792ZINFOtest_replacement_sanity (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
88782025-04-30T06:25:55.792ZDEBGtest_replacement_sanity (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
88792025-04-30T06:25:55.793ZDEBGtest_replacement_sanity (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
88802025-04-30T06:25:55.793ZWARNtest_replacement_sanity (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
88812025-04-30T06:25:55.795ZWARNtest_replacement_sanity (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
88822025-04-30T06:25:55.795ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3220, tv_nsec: 594469630 } })
88832025-04-30T06:25:55.795ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
88842025-04-30T06:25:55.795ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
88852025-04-30T06:25:55.796ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
88862025-04-30T06:25:55.796ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
88872025-04-30T06:25:55.806ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
88882025-04-30T06:25:55.807ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
88892025-04-30T06:25:55.809ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
88902025-04-30T06:25:55.809ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88912025-04-30T06:25:55.809ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
88922025-04-30T06:25:55.809ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
88932025-04-30T06:25:55.811ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
88942025-04-30T06:25:55.812ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 237817 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 2526fe3e-fc0f-4c28-aba8-fd6b1fc746b8 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
88952025-04-30T06:25:55.817ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "2526fe3e-fc0f-4c28-aba8-fd6b1fc746b8", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:55 GMT"} })
88962025-04-30T06:25:55.817ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
88972025-04-30T06:25:55.818ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
88982025-04-30T06:25:55.818ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
88992025-04-30T06:25:55.818ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "79284e80-66f3-4c9d-8834-e60051361e91", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:55 GMT"} })
89002025-04-30T06:25:55.818ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
89012025-04-30T06:25:55.820ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89022025-04-30T06:25:55.825ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
89032025-04-30T06:25:55.825ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
89042025-04-30T06:25:55.825ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89052025-04-30T06:25:55.825ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
89062025-04-30T06:25:55.825ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89072025-04-30T06:25:55.826ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89082025-04-30T06:25:55.828ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
89092025-04-30T06:25:55.828ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
89102025-04-30T06:25:55.831ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d3 3b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .;..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56529 req_id = d154dafe-d27a-47c8-b2e1-13197b955f32
89112025-04-30T06:25:55.831ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54075,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56529 req_id = d154dafe-d27a-47c8-b2e1-13197b955f32
89122025-04-30T06:25:55.831ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
89132025-04-30T06:25:55.832ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56529 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d154dafe-d27a-47c8-b2e1-13197b955f32
89142025-04-30T06:25:55.832ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd f5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62114 req_id = e581ea26-eb78-4359-9524-7162a5b7bfd0
89152025-04-30T06:25:55.832ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48629,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62114 req_id = e581ea26-eb78-4359-9524-7162a5b7bfd0
89162025-04-30T06:25:55.832ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89172025-04-30T06:25:55.832ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62114 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e581ea26-eb78-4359-9524-7162a5b7bfd0
89182025-04-30T06:25:55.837ZINFOtest_replacement_sanity (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
89192025-04-30T06:25:55.895ZINFOtest_replacement_sanity (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
89202025-04-30T06:25:55.902ZDEBGtest_replacement_sanity (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
89212025-04-30T06:25:55.902ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89222025-04-30T06:25:55.912ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
89232025-04-30T06:25:55.914ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
89242025-04-30T06:25:55.917ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
89252025-04-30T06:25:55.918ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
89262025-04-30T06:25:55.918ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
89272025-04-30T06:25:55.918ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89282025-04-30T06:25:55.922ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
89292025-04-30T06:25:56.195ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 039be560-54cc-49e3-88df-1a29dadbf913 background_task = v2p_manager body = None method = GET uri = http://[::1]:42668/v2p
89302025-04-30T06:25:56.516ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager sled 039be560-54cc-49e3-88df-1a29dadbf913 background_task = vpc_route_manager
89312025-04-30T06:25:56.517ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 039be560-54cc-49e3-88df-1a29dadbf913 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:42668/vpc-routes
89322025-04-30T06:25:56.518ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:58730 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
89332025-04-30T06:25:56.518ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:60537 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
89342025-04-30T06:25:56.519ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89352025-04-30T06:25:56.523ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89362025-04-30T06:25:56.528ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:58730 req_id = 21e60b35-6e8e-433d-a368-03f46faa3827 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /v2p
89372025-04-30T06:25:56.528ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:60537 req_id = 14f880fe-7ee0-4a23-8a95-48c722510b79 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /vpc-routes
89382025-04-30T06:25:56.529ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bad01d0f-c944-489e-a3b9-e6a02f4ab031", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
89392025-04-30T06:25:56.529ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
89402025-04-30T06:25:56.531ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3048 local_addr = [::1]:42668 method = GET remote_addr = [::1]:58730 req_id = 21e60b35-6e8e-433d-a368-03f46faa3827 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /v2p
89412025-04-30T06:25:56.531ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3002 local_addr = [::1]:42668 method = GET remote_addr = [::1]:60537 req_id = 14f880fe-7ee0-4a23-8a95-48c722510b79 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /vpc-routes
89422025-04-30T06:25:56.547ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 039be560-54cc-49e3-88df-1a29dadbf913 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:42668/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "21e60b35-6e8e-433d-a368-03f46faa3827", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
89432025-04-30T06:25:56.547ZINFOtest_replacement_sanity (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-039be560
89442025-04-30T06:25:56.547ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-039be560
89452025-04-30T06:25:56.547ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-039be560
89462025-04-30T06:25:56.550ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = v2p_manager body = None method = GET uri = http://[::1]:47918/v2p
89472025-04-30T06:25:56.550ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89482025-04-30T06:25:56.552ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89492025-04-30T06:25:56.552ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89502025-04-30T06:25:56.552ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89512025-04-30T06:25:56.552ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 039be560-54cc-49e3-88df-1a29dadbf913 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:42668/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "14f880fe-7ee0-4a23-8a95-48c722510b79", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
89522025-04-30T06:25:56.552ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
89532025-04-30T06:25:56.552ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
89542025-04-30T06:25:56.553ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
89552025-04-30T06:25:56.558ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:49403 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
89562025-04-30T06:25:56.560ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
89572025-04-30T06:25:56.561ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:49403 req_id = e0cfcef0-c9af-4aca-87ca-89143066e145 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /v2p
89582025-04-30T06:25:56.579ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 18278 local_addr = [::1]:47918 method = GET remote_addr = [::1]:49403 req_id = e0cfcef0-c9af-4aca-87ca-89143066e145 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /v2p
89592025-04-30T06:25:56.581ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = v2p_manager result = Ok(Response { url: "http://[::1]:47918/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0cfcef0-c9af-4aca-87ca-89143066e145", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
89602025-04-30T06:25:56.581ZINFOtest_replacement_sanity (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-2823225f
89612025-04-30T06:25:56.581ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-2823225f
89622025-04-30T06:25:56.581ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-2823225f
89632025-04-30T06:25:56.582ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = v2p_manager body = None method = GET uri = http://[::1]:35744/v2p
89642025-04-30T06:25:56.583ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
89652025-04-30T06:25:56.583ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89662025-04-30T06:25:56.583ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
89672025-04-30T06:25:56.584ZINFOtest_replacement_sanity (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-039be560
89682025-04-30T06:25:56.584ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 039be560-54cc-49e3-88df-1a29dadbf913 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:42668/eip-gateways
89692025-04-30T06:25:56.584ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:55350 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
89702025-04-30T06:25:56.585ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = PUT remote_addr = [::1]:60537 req_id = 4ddb8119-19ba-4c50-8840-d7c1a077d312 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /eip-gateways
89712025-04-30T06:25:56.591ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 6403 local_addr = [::1]:42668 method = PUT remote_addr = [::1]:60537 req_id = 4ddb8119-19ba-4c50-8840-d7c1a077d312 response_code = 204 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /eip-gateways
89722025-04-30T06:25:56.592ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:55350 req_id = 89ef4439-cfb6-401e-b034-4f237d0906a3 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /v2p
89732025-04-30T06:25:56.592ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 039be560-54cc-49e3-88df-1a29dadbf913 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:42668/eip-gateways", status: 204, headers: {"x-request-id": "4ddb8119-19ba-4c50-8840-d7c1a077d312", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
89742025-04-30T06:25:56.594ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager sled 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = vpc_route_manager
89752025-04-30T06:25:56.594ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = vpc_route_manager body = None method = GET uri = http://[::1]:47918/vpc-routes
89762025-04-30T06:25:56.594ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 2428 local_addr = [::1]:35744 method = GET remote_addr = [::1]:55350 req_id = 89ef4439-cfb6-401e-b034-4f237d0906a3 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /v2p
89772025-04-30T06:25:56.595ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:45644 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
89782025-04-30T06:25:56.608ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = v2p_manager result = Ok(Response { url: "http://[::1]:35744/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ef4439-cfb6-401e-b034-4f237d0906a3", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
89792025-04-30T06:25:56.608ZINFOtest_replacement_sanity (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-98fd4d7a
89802025-04-30T06:25:56.608ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-98fd4d7a
89812025-04-30T06:25:56.608ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-98fd4d7a
89822025-04-30T06:25:56.610ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:35343/v2p
89832025-04-30T06:25:56.611ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89842025-04-30T06:25:56.611ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89852025-04-30T06:25:56.611ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89862025-04-30T06:25:56.611ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
89872025-04-30T06:25:56.612ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89882025-04-30T06:25:56.613ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:62114 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
89892025-04-30T06:25:56.613ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:45644 req_id = 0164795c-2743-44e6-a7d2-a15fe9c82cb2 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /vpc-routes
89902025-04-30T06:25:56.613ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89912025-04-30T06:25:56.613ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
89922025-04-30T06:25:56.614ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 821 local_addr = [::1]:47918 method = GET remote_addr = [::1]:45644 req_id = 0164795c-2743-44e6-a7d2-a15fe9c82cb2 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /vpc-routes
89932025-04-30T06:25:56.614ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
89942025-04-30T06:25:56.615ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
89952025-04-30T06:25:56.615ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 039be560-54cc-49e3-88df-1a29dadbf913, lookup_type: ById(039be560-54cc-49e3-88df-1a29dadbf913) }
89962025-04-30T06:25:56.615ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89972025-04-30T06:25:56.615ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
89982025-04-30T06:25:56.615ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
89992025-04-30T06:25:56.615ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90002025-04-30T06:25:56.618ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:62114 req_id = f038fef5-bc50-4b47-88f4-d064919834ac sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
90012025-04-30T06:25:56.624ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:47918/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "0164795c-2743-44e6-a7d2-a15fe9c82cb2", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90022025-04-30T06:25:56.625ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
90032025-04-30T06:25:56.625ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90042025-04-30T06:25:56.625ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90052025-04-30T06:25:56.626ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 8198 local_addr = [::1]:35343 method = GET remote_addr = [::1]:62114 req_id = f038fef5-bc50-4b47-88f4-d064919834ac response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
90062025-04-30T06:25:56.626ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90072025-04-30T06:25:56.631ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90082025-04-30T06:25:56.632ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90092025-04-30T06:25:56.632ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
90102025-04-30T06:25:56.632ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90112025-04-30T06:25:56.634ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
90122025-04-30T06:25:56.634ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
90132025-04-30T06:25:56.635ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:35343/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "f038fef5-bc50-4b47-88f4-d064919834ac", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90142025-04-30T06:25:56.635ZINFOtest_replacement_sanity (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
90152025-04-30T06:25:56.635ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
90162025-04-30T06:25:56.635ZINFOtest_replacement_sanity (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
90172025-04-30T06:25:56.637ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 1.198542197s iteration = 2 status = {}
90182025-04-30T06:25:56.638ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90192025-04-30T06:25:56.644ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 039be560-54cc-49e3-88df-1a29dadbf913, lookup_type: ById(039be560-54cc-49e3-88df-1a29dadbf913) } result = Ok(())
90202025-04-30T06:25:56.644ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
90212025-04-30T06:25:56.644ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90222025-04-30T06:25:56.644ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90232025-04-30T06:25:56.654ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2823225f-fa9c-49ed-b0f5-744d993ad37d, lookup_type: ById(2823225f-fa9c-49ed-b0f5-744d993ad37d) }
90242025-04-30T06:25:56.654ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90252025-04-30T06:25:56.654ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
90262025-04-30T06:25:56.654ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90272025-04-30T06:25:56.655ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90282025-04-30T06:25:56.657ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90292025-04-30T06:25:56.657ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
90302025-04-30T06:25:56.657ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90312025-04-30T06:25:56.659ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
90322025-04-30T06:25:56.659ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
90332025-04-30T06:25:56.660ZINFOtest_replacement_sanity (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-2823225f
90342025-04-30T06:25:56.660ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:47918/eip-gateways
90352025-04-30T06:25:56.660ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = PUT remote_addr = [::1]:45644 req_id = c8f5ebc6-cbd4-4cb7-a686-61face06305a sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /eip-gateways
90362025-04-30T06:25:56.668ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 7060 local_addr = [::1]:47918 method = PUT remote_addr = [::1]:45644 req_id = c8f5ebc6-cbd4-4cb7-a686-61face06305a response_code = 204 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /eip-gateways
90372025-04-30T06:25:56.668ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90382025-04-30T06:25:56.676ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 2823225f-fa9c-49ed-b0f5-744d993ad37d, lookup_type: ById(2823225f-fa9c-49ed-b0f5-744d993ad37d) } result = Ok(())
90392025-04-30T06:25:56.676ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
90402025-04-30T06:25:56.676ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90412025-04-30T06:25:56.676ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90422025-04-30T06:25:56.678ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 2823225f-fa9c-49ed-b0f5-744d993ad37d background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:47918/eip-gateways", status: 204, headers: {"x-request-id": "c8f5ebc6-cbd4-4cb7-a686-61face06305a", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90432025-04-30T06:25:56.680ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager sled 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = vpc_route_manager
90442025-04-30T06:25:56.680ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = vpc_route_manager body = None method = GET uri = http://[::1]:35744/vpc-routes
90452025-04-30T06:25:56.685ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:38147 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
90462025-04-30T06:25:56.690ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da, lookup_type: ById(98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da) }
90472025-04-30T06:25:56.690ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90482025-04-30T06:25:56.690ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
90492025-04-30T06:25:56.690ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90502025-04-30T06:25:56.690ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90512025-04-30T06:25:56.693ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:38147 req_id = 7315b1b2-b981-419b-a60e-ab751c99eedb sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /vpc-routes
90522025-04-30T06:25:56.694ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1531 local_addr = [::1]:35744 method = GET remote_addr = [::1]:38147 req_id = 7315b1b2-b981-419b-a60e-ab751c99eedb response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /vpc-routes
90532025-04-30T06:25:56.699ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:35744/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "7315b1b2-b981-419b-a60e-ab751c99eedb", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90542025-04-30T06:25:56.699ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
90552025-04-30T06:25:56.699ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90562025-04-30T06:25:56.701ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90572025-04-30T06:25:56.702ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90582025-04-30T06:25:56.707ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da, lookup_type: ById(98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da) } result = Ok(())
90592025-04-30T06:25:56.708ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
90602025-04-30T06:25:56.708ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90612025-04-30T06:25:56.708ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90622025-04-30T06:25:56.714ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de8139c5-dc28-4a9f-9a5c-b8c71905709c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90632025-04-30T06:25:56.714ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
90642025-04-30T06:25:56.723ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
90652025-04-30T06:25:56.724ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
90662025-04-30T06:25:56.724ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90672025-04-30T06:25:56.724ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
90682025-04-30T06:25:56.724ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
90692025-04-30T06:25:56.725ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
90702025-04-30T06:25:56.729ZINFOtest_replacement_sanity (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-98fd4d7a
90712025-04-30T06:25:56.729ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:35744/eip-gateways
90722025-04-30T06:25:56.729ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = PUT remote_addr = [::1]:38147 req_id = d8986de9-bf8c-4c16-817b-5c4b3052af6e sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /eip-gateways
90732025-04-30T06:25:56.731ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1531 local_addr = [::1]:35744 method = PUT remote_addr = [::1]:38147 req_id = d8986de9-bf8c-4c16-817b-5c4b3052af6e response_code = 204 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /eip-gateways
90742025-04-30T06:25:56.731ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90752025-04-30T06:25:56.737ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
90762025-04-30T06:25:56.737ZERROtest_replacement_sanity (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
90772025-04-30T06:25:56.737ZERROtest_replacement_sanity (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
90782025-04-30T06:25:56.737ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 1.299899004s iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
90792025-04-30T06:25:56.738ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:35744/eip-gateways", status: 204, headers: {"x-request-id": "d8986de9-bf8c-4c16-817b-5c4b3052af6e", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90802025-04-30T06:25:56.739ZINFOtest_replacement_sanity (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
90812025-04-30T06:25:56.740ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:35343/vpc-routes
90822025-04-30T06:25:56.742ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:48889 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
90832025-04-30T06:25:56.742ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:48889 req_id = e26deb35-5177-4661-b5d3-a3b7d2d35a75 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
90842025-04-30T06:25:56.742ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 169 local_addr = [::1]:35343 method = GET remote_addr = [::1]:48889 req_id = e26deb35-5177-4661-b5d3-a3b7d2d35a75 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
90852025-04-30T06:25:56.743ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:35343/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26deb35-5177-4661-b5d3-a3b7d2d35a75", "content-length": "2", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90862025-04-30T06:25:56.743ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
90872025-04-30T06:25:56.743ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
90882025-04-30T06:25:56.743ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
90892025-04-30T06:25:56.747ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1289ab3-c4e6-492d-b741-73885584eef6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90902025-04-30T06:25:56.747ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
90912025-04-30T06:25:56.749ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15929d15-99ca-48ba-8df0-56f2a19ae64f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90922025-04-30T06:25:56.749ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
90932025-04-30T06:25:56.751ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d0d13af-dddc-474a-92f5-744c7f2e0659", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90942025-04-30T06:25:56.751ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
90952025-04-30T06:25:56.754ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
90962025-04-30T06:25:56.755ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c40903f0-a161-4a63-8a2f-53de3b648487", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90972025-04-30T06:25:56.756ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
90982025-04-30T06:25:56.757ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "452c23e4-dcc7-4fe3-aa0c-d31d144ac525", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
90992025-04-30T06:25:56.758ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
91002025-04-30T06:25:56.758ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f5e2cfc-e0f2-4708-b126-5db944e92434", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91012025-04-30T06:25:56.759ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
91022025-04-30T06:25:56.759ZINFOtest_replacement_sanity (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
91032025-04-30T06:25:56.759ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:35343/eip-gateways
91042025-04-30T06:25:56.760ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = PUT remote_addr = [::1]:48889 req_id = 449d77b6-bdbc-4561-9902-bb4c1dd8144f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
91052025-04-30T06:25:56.760ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 266 local_addr = [::1]:35343 method = PUT remote_addr = [::1]:48889 req_id = 449d77b6-bdbc-4561-9902-bb4c1dd8144f response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
91062025-04-30T06:25:56.760ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:35343/eip-gateways", status: 204, headers: {"x-request-id": "449d77b6-bdbc-4561-9902-bb4c1dd8144f", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91072025-04-30T06:25:56.762ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 1.319876527s iteration = 2 status = {}
91082025-04-30T06:25:56.763ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16bbe9e0-2419-4785-bb4c-2e7d0a8c7e0a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91092025-04-30T06:25:56.763ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
91102025-04-30T06:25:56.764ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2626e150-33e9-4c77-8474-ce9cee0d070f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91112025-04-30T06:25:56.764ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
91122025-04-30T06:25:56.765ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90d98f5-4a46-4730-8120-fa82ca629461", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91132025-04-30T06:25:56.765ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
91142025-04-30T06:25:56.767ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed72de0-c652-4937-acb0-80c7d52fd180", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91152025-04-30T06:25:56.767ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
91162025-04-30T06:25:56.769ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "57917efc-4d50-4580-b9b8-0fa21995411a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91172025-04-30T06:25:56.769ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
91182025-04-30T06:25:56.775ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9018c3ec-569c-47c7-9b14-f25781ec254e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91192025-04-30T06:25:56.775ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
91202025-04-30T06:25:56.777ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc03dd4f-0260-4a4e-8932-0effedfa3153", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91212025-04-30T06:25:56.778ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
91222025-04-30T06:25:56.779ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f1813da-ca10-4d5e-a43b-d254b0d105c2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91232025-04-30T06:25:56.780ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
91242025-04-30T06:25:56.782ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e39ceb5-398e-4ce0-afa8-7d3ae20ec493", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91252025-04-30T06:25:56.782ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
91262025-04-30T06:25:56.784ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b1c2fbc-4531-4035-b7d3-a105d7af01bf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91272025-04-30T06:25:56.785ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
91282025-04-30T06:25:56.787ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17053adf-68e8-4d26-9a73-b1845f27ca44", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91292025-04-30T06:25:56.787ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
91302025-04-30T06:25:56.789ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95b0f3bd-ea8e-404b-bd9d-fe0d0fb451c9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91312025-04-30T06:25:56.789ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
91322025-04-30T06:25:56.793ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "311663cb-b314-41cd-9284-ffb871c58cc8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91332025-04-30T06:25:56.794ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
91342025-04-30T06:25:56.795ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9712474-d46f-44cb-b16b-26cc0b88dc06", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91352025-04-30T06:25:56.795ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
91362025-04-30T06:25:56.800ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a987ac8-8571-4273-8572-8d2f1cea8cce", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91372025-04-30T06:25:56.800ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
91382025-04-30T06:25:56.802ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1678dcaf-11eb-42d4-89b2-e47b95950861", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91392025-04-30T06:25:56.802ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
91402025-04-30T06:25:56.804ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b355d7-5b8b-40e9-b769-4ffc6f89a93d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91412025-04-30T06:25:56.804ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
91422025-04-30T06:25:56.805ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91432025-04-30T06:25:56.805ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91442025-04-30T06:25:56.805ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91452025-04-30T06:25:56.805ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
91462025-04-30T06:25:56.806ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91472025-04-30T06:25:56.810ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "00c0be26-4240-43f2-a11e-57ff8d0ab29c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91482025-04-30T06:25:56.810ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
91492025-04-30T06:25:56.811ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91502025-04-30T06:25:56.818ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91512025-04-30T06:25:56.818ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91522025-04-30T06:25:56.818ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
91532025-04-30T06:25:56.819ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91542025-04-30T06:25:56.822ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91552025-04-30T06:25:56.822ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
91562025-04-30T06:25:56.823ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91572025-04-30T06:25:56.825ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "83170f83-4b64-400e-bd2b-cd42d373e0a9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91582025-04-30T06:25:56.825ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
91592025-04-30T06:25:56.829ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad941e77-5e7c-4c61-9602-c259a674a2d4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91602025-04-30T06:25:56.829ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
91612025-04-30T06:25:56.833ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23f2beb7-aa73-4988-969f-1dd25f7ab9a4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91622025-04-30T06:25:56.833ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
91632025-04-30T06:25:56.835ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f293245-1bb3-48c2-8b0b-2f30fffd2959", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91642025-04-30T06:25:56.835ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
91652025-04-30T06:25:56.837ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "874a65cd-cba6-40bd-92a2-e1a2fb1f1c3f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91662025-04-30T06:25:56.837ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
91672025-04-30T06:25:56.839ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "deced73a-b6a5-4fb7-bde0-ac074e99a878", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91682025-04-30T06:25:56.839ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
91692025-04-30T06:25:56.842ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef6bc9d3-da62-4dd3-a23d-d66fc529950c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91702025-04-30T06:25:56.842ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
91712025-04-30T06:25:56.844ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dfe7fc7-78d1-459f-b644-c1c912dcdc65", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91722025-04-30T06:25:56.844ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
91732025-04-30T06:25:56.846ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "653e2836-f86c-4fcb-b1e7-c54a84ac110a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91742025-04-30T06:25:56.846ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
91752025-04-30T06:25:56.852ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5005aaf-f202-4e94-b564-389502fa8219", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91762025-04-30T06:25:56.852ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
91772025-04-30T06:25:56.852ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
91782025-04-30T06:25:56.855ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36e3febf-478a-43ff-b758-0007e439542d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91792025-04-30T06:25:56.855ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
91802025-04-30T06:25:56.858ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31f46be5-52a9-40d4-8d6f-24bdff947b76", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91812025-04-30T06:25:56.858ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
91822025-04-30T06:25:56.862ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "130a2246-1789-4962-a6aa-f8e22ea34b7e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91832025-04-30T06:25:56.862ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
91842025-04-30T06:25:56.865ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0eba1f3-367c-4d25-8296-9c06ec4ec994", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91852025-04-30T06:25:56.865ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
91862025-04-30T06:25:56.866ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a6a4f7f-5e7a-4465-a94c-6b15c67ca098", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91872025-04-30T06:25:56.866ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
91882025-04-30T06:25:56.868ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26fa3b87-e0c1-4d2e-8da9-4cf28549a5e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91892025-04-30T06:25:56.869ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
91902025-04-30T06:25:56.871ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9871330e-040f-4984-ad82-78c96ef4e473", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91912025-04-30T06:25:56.871ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
91922025-04-30T06:25:56.873ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0215636-d815-4282-b9f6-6b564e754e79", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91932025-04-30T06:25:56.873ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
91942025-04-30T06:25:56.873ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
91952025-04-30T06:25:56.877ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "248a302c-7042-4a00-8223-31e23832a74b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91962025-04-30T06:25:56.877ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
91972025-04-30T06:25:56.878ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bc8c6f4-200e-4d1c-a4cf-c8047a3d6213", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
91982025-04-30T06:25:56.879ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
91992025-04-30T06:25:56.880ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b448a43-4350-420c-b561-e25a9b6bb76f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92002025-04-30T06:25:56.880ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
92012025-04-30T06:25:56.881ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f87279e1-51e1-4c3c-a510-1e4625a40eb0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92022025-04-30T06:25:56.881ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
92032025-04-30T06:25:56.882ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d0c6790-0168-4c4b-b5fd-5120f9f52562", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92042025-04-30T06:25:56.884ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92052025-04-30T06:25:56.884ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92062025-04-30T06:25:56.884ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92072025-04-30T06:25:56.884ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92082025-04-30T06:25:56.885ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92092025-04-30T06:25:56.887ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92102025-04-30T06:25:56.891ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92112025-04-30T06:25:56.891ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
92122025-04-30T06:25:56.891ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92132025-04-30T06:25:56.891ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92142025-04-30T06:25:56.891ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92152025-04-30T06:25:56.892ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92162025-04-30T06:25:56.904ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92172025-04-30T06:25:56.905ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92182025-04-30T06:25:56.905ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92192025-04-30T06:25:56.905ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
92202025-04-30T06:25:56.909ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92212025-04-30T06:25:56.909ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92222025-04-30T06:25:56.914ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
92232025-04-30T06:25:56.914ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92242025-04-30T06:25:56.914ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92252025-04-30T06:25:56.915ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92262025-04-30T06:25:56.917ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92272025-04-30T06:25:56.924ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92282025-04-30T06:25:56.924ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92292025-04-30T06:25:56.924ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
92302025-04-30T06:25:56.925ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92312025-04-30T06:25:56.986ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
92322025-04-30T06:25:56.986ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
92332025-04-30T06:25:56.987ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:42977 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
92342025-04-30T06:25:56.987ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:42977 req_id = 92416dac-99a8-4edc-bab5-c87368395e88 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
92352025-04-30T06:25:56.988ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 323 local_addr = [::1]:42668 method = GET remote_addr = [::1]:42977 req_id = 92416dac-99a8-4edc-bab5-c87368395e88 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
92362025-04-30T06:25:56.988ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "92416dac-99a8-4edc-bab5-c87368395e88", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92372025-04-30T06:25:57.035ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
92382025-04-30T06:25:57.035ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
92392025-04-30T06:25:57.035ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92402025-04-30T06:25:57.035ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
92412025-04-30T06:25:57.036ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92422025-04-30T06:25:57.036ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:34983 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
92432025-04-30T06:25:57.037ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:34983 req_id = 5c3d02f7-2f76-454d-b2ea-a0422c9e62f9 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
92442025-04-30T06:25:57.037ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 264 local_addr = [::1]:47918 method = GET remote_addr = [::1]:34983 req_id = 5c3d02f7-2f76-454d-b2ea-a0422c9e62f9 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
92452025-04-30T06:25:57.037ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c3d02f7-2f76-454d-b2ea-a0422c9e62f9", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92462025-04-30T06:25:57.087ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
92472025-04-30T06:25:57.088ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
92482025-04-30T06:25:57.089ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:38964 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
92492025-04-30T06:25:57.093ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:38964 req_id = 971de674-c00d-42da-bcbb-df19a0e0b5db sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
92502025-04-30T06:25:57.093ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 533 local_addr = [::1]:35744 method = GET remote_addr = [::1]:38964 req_id = 971de674-c00d-42da-bcbb-df19a0e0b5db response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
92512025-04-30T06:25:57.095ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "971de674-c00d-42da-bcbb-df19a0e0b5db", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92522025-04-30T06:25:57.153ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
92532025-04-30T06:25:57.153ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
92542025-04-30T06:25:57.156ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:46038 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
92552025-04-30T06:25:57.158ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:46038 req_id = 3489fba8-4446-4e24-b8ca-9dec9c4ea171 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92562025-04-30T06:25:57.158ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 530 local_addr = [::1]:35343 method = GET remote_addr = [::1]:46038 req_id = 3489fba8-4446-4e24-b8ca-9dec9c4ea171 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
92572025-04-30T06:25:57.159ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3489fba8-4446-4e24-b8ca-9dec9c4ea171", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
92582025-04-30T06:25:57.162ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92592025-04-30T06:25:57.162ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
92602025-04-30T06:25:57.162ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
92612025-04-30T06:25:57.168ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92622025-04-30T06:25:57.168ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92632025-04-30T06:25:57.168ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92642025-04-30T06:25:57.168ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92652025-04-30T06:25:57.169ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92662025-04-30T06:25:57.178ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92672025-04-30T06:25:57.182ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92682025-04-30T06:25:57.182ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92692025-04-30T06:25:57.182ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92702025-04-30T06:25:57.183ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92712025-04-30T06:25:57.237ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
92722025-04-30T06:25:57.251ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
92732025-04-30T06:25:57.251ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
92742025-04-30T06:25:57.251ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 2.007805732s iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
92752025-04-30T06:25:57.251ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
92762025-04-30T06:25:57.252ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
92772025-04-30T06:25:57.252ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92782025-04-30T06:25:57.252ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92792025-04-30T06:25:57.252ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92802025-04-30T06:25:57.253ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92812025-04-30T06:25:57.256ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92822025-04-30T06:25:57.265ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
92832025-04-30T06:25:57.265ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
92842025-04-30T06:25:57.265ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
92852025-04-30T06:25:57.265ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
92862025-04-30T06:25:57.271ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
92872025-04-30T06:25:57.272ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
92882025-04-30T06:25:57.273ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c5 6a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .j..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51758 req_id = d3c3c464-513e-4ef1-b4eb-f1385484a6b1
92892025-04-30T06:25:57.274ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50538,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51758 req_id = d3c3c464-513e-4ef1-b4eb-f1385484a6b1
92902025-04-30T06:25:57.274ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
92912025-04-30T06:25:57.274ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
92922025-04-30T06:25:57.274ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92932025-04-30T06:25:57.274ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51758 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d3c3c464-513e-4ef1-b4eb-f1385484a6b1
92942025-04-30T06:25:57.275ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3222, tv_nsec: 74333883 } })
92952025-04-30T06:25:57.275ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
92962025-04-30T06:25:57.277ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
92972025-04-30T06:25:57.278ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 69 80 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e i...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53857 req_id = 621a9da0-dd06-4013-bff9-f1f829914efc
92982025-04-30T06:25:57.278ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27008,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53857 req_id = 621a9da0-dd06-4013-bff9-f1f829914efc
92992025-04-30T06:25:57.278ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
93002025-04-30T06:25:57.278ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53857 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 621a9da0-dd06-4013-bff9-f1f829914efc
93012025-04-30T06:25:57.278ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6f 5d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e o]..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49300 req_id = b817f328-30b0-46fe-9c82-cecbce8737d3
93022025-04-30T06:25:57.278ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28509,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49300 req_id = b817f328-30b0-46fe-9c82-cecbce8737d3
93032025-04-30T06:25:57.278ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93042025-04-30T06:25:57.278ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49300 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b817f328-30b0-46fe-9c82-cecbce8737d3
93052025-04-30T06:25:57.414ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
93062025-04-30T06:25:57.415ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93072025-04-30T06:25:57.415ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93082025-04-30T06:25:57.415ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93092025-04-30T06:25:57.415ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
93102025-04-30T06:25:57.417ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93112025-04-30T06:25:57.417ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
93122025-04-30T06:25:57.417ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93132025-04-30T06:25:57.417ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93142025-04-30T06:25:57.418ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93152025-04-30T06:25:57.418ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b3adc29e-97f9-48c3-997e-f8caf0bc4529 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93162025-04-30T06:25:57.419ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: aa d2 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64510 req_id = 7c29acd2-b935-4d8e-9640-55041340e33f
93172025-04-30T06:25:57.419ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43730,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64510 req_id = 7c29acd2-b935-4d8e-9640-55041340e33f
93182025-04-30T06:25:57.419ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
93192025-04-30T06:25:57.419ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 43730, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64510 req_id = 7c29acd2-b935-4d8e-9640-55041340e33f
93202025-04-30T06:25:57.419ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b3adc29e-97f9-48c3-997e-f8caf0bc4529 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93212025-04-30T06:25:57.420ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b3adc29e-97f9-48c3-997e-f8caf0bc4529 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93222025-04-30T06:25:57.420ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b3adc29e-97f9-48c3-997e-f8caf0bc4529 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93232025-04-30T06:25:57.423ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
93242025-04-30T06:25:57.423ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
93252025-04-30T06:25:57.423ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
93262025-04-30T06:25:57.423ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93272025-04-30T06:25:57.424ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93282025-04-30T06:25:57.424ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93292025-04-30T06:25:57.424ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93302025-04-30T06:25:57.425ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93312025-04-30T06:25:57.432ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93322025-04-30T06:25:57.432ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93332025-04-30T06:25:57.432ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
93342025-04-30T06:25:57.433ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93352025-04-30T06:25:57.435ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 16674 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = b3adc29e-97f9-48c3-997e-f8caf0bc4529 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
93362025-04-30T06:25:57.436ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3adc29e-97f9-48c3-997e-f8caf0bc4529", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93372025-04-30T06:25:57.436ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
93382025-04-30T06:25:57.436ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93392025-04-30T06:25:57.436ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
93402025-04-30T06:25:57.438ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93412025-04-30T06:25:57.438ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = d11357c5-1d96-458d-afea-7b5b29ae18eb uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
93422025-04-30T06:25:57.439ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = d11357c5-1d96-458d-afea-7b5b29ae18eb resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
93432025-04-30T06:25:57.439ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = d11357c5-1d96-458d-afea-7b5b29ae18eb roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
93442025-04-30T06:25:57.439ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = d11357c5-1d96-458d-afea-7b5b29ae18eb resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
93452025-04-30T06:25:57.446ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e892c76-7012-4f74-9c7d-bc03ab5e0780", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93462025-04-30T06:25:57.446ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
93472025-04-30T06:25:57.452ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 13453 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = d11357c5-1d96-458d-afea-7b5b29ae18eb response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
93482025-04-30T06:25:57.453ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d11357c5-1d96-458d-afea-7b5b29ae18eb", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93492025-04-30T06:25:57.453ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
93502025-04-30T06:25:57.453ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
93512025-04-30T06:25:57.453ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
93522025-04-30T06:25:57.465ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85588a72-5001-4c4c-a455-c3229db3e963", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93532025-04-30T06:25:57.465ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
93542025-04-30T06:25:57.474ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "124bb00c-032a-44e9-82ec-e5610d5b357d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93552025-04-30T06:25:57.474ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
93562025-04-30T06:25:57.477ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4476bd5b-efba-4ea9-a5a1-14e074ccb276", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93572025-04-30T06:25:57.477ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
93582025-04-30T06:25:57.482ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25715c97-8c99-4ca6-88a7-cee7a20cf393", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93592025-04-30T06:25:57.482ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
93602025-04-30T06:25:57.493ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c78c176-3cbc-4ecd-8343-7464c731dde5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93612025-04-30T06:25:57.493ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
93622025-04-30T06:25:57.497ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
93632025-04-30T06:25:57.500ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "226a1184-a70f-456f-b142-72998a6359a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93642025-04-30T06:25:57.500ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
93652025-04-30T06:25:57.505ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc02ae00-15cf-408c-b066-caffd449b712", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93662025-04-30T06:25:57.505ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
93672025-04-30T06:25:57.506ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "276dcc20-6853-4692-a420-841a8a3ff523", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93682025-04-30T06:25:57.507ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
93692025-04-30T06:25:57.509ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f151a76e-a619-4656-830c-5051b0aa1ae6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93702025-04-30T06:25:57.509ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
93712025-04-30T06:25:57.511ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "103dc860-b933-4cab-91c6-ff522167323c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93722025-04-30T06:25:57.511ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
93732025-04-30T06:25:57.512ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "773ca0b1-c41a-46dc-a779-3c35a71825be", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93742025-04-30T06:25:57.512ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
93752025-04-30T06:25:57.515ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1516c3a2-b918-413b-bceb-9930367d328c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93762025-04-30T06:25:57.515ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
93772025-04-30T06:25:57.517ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df3c9c4b-22e8-4ba2-9e76-f9b2288fc093", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93782025-04-30T06:25:57.518ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
93792025-04-30T06:25:57.519ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c49b5a19-3f34-4195-b874-bb5ed55d6fe1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93802025-04-30T06:25:57.520ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
93812025-04-30T06:25:57.522ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb709079-eeea-4fc6-a44f-a44b5c6ce248", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:56 GMT"} })
93822025-04-30T06:25:57.522ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
93832025-04-30T06:25:57.525ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38043b9b-4664-4815-aa31-747f3c43be7a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93842025-04-30T06:25:57.525ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
93852025-04-30T06:25:57.529ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e2be40d-41c6-4487-b4fb-54c5e13b6e42", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93862025-04-30T06:25:57.529ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
93872025-04-30T06:25:57.531ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "198e7e90-5b22-4109-8cfb-8dc3e3322bcd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93882025-04-30T06:25:57.531ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
93892025-04-30T06:25:57.532ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2861c91d-366a-4d2c-bcb2-266819104785", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93902025-04-30T06:25:57.532ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
93912025-04-30T06:25:57.534ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20572d25-8220-4845-900e-8778c1081f1f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93922025-04-30T06:25:57.534ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
93932025-04-30T06:25:57.536ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8616532f-d03c-4164-b488-e6a40e8073e0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93942025-04-30T06:25:57.536ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
93952025-04-30T06:25:57.538ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "add4a98f-d3dd-419c-a767-903c70cf96cd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93962025-04-30T06:25:57.538ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
93972025-04-30T06:25:57.539ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ec945f8-62eb-4bd5-98da-3c7bfff26fae", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
93982025-04-30T06:25:57.539ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
93992025-04-30T06:25:57.542ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de316392-f612-4d38-8b74-f8b9ca3b9914", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94002025-04-30T06:25:57.542ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
94012025-04-30T06:25:57.545ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "52fc9c09-854e-4efb-a6f7-bc62c2506f5b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94022025-04-30T06:25:57.545ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
94032025-04-30T06:25:57.547ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6263066-5779-4ec1-990d-5bb086c0dfc0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94042025-04-30T06:25:57.548ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
94052025-04-30T06:25:57.548ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94062025-04-30T06:25:57.548ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94072025-04-30T06:25:57.548ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94082025-04-30T06:25:57.548ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
94092025-04-30T06:25:57.549ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94102025-04-30T06:25:57.552ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2165795-ad13-4777-946e-468aec972886", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94112025-04-30T06:25:57.552ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
94122025-04-30T06:25:57.553ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94132025-04-30T06:25:57.559ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94142025-04-30T06:25:57.559ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94152025-04-30T06:25:57.559ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
94162025-04-30T06:25:57.560ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94172025-04-30T06:25:57.565ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94182025-04-30T06:25:57.565ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
94192025-04-30T06:25:57.566ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94202025-04-30T06:25:57.568ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9533a7d-b07b-4470-9d86-3522212d58f3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94212025-04-30T06:25:57.568ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
94222025-04-30T06:25:57.571ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8d1e935-1c57-4d9d-86cc-f071381b6c91", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94232025-04-30T06:25:57.571ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
94242025-04-30T06:25:57.572ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7533eb9-20a7-4f41-8b6a-9d86c99c2fef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94252025-04-30T06:25:57.572ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
94262025-04-30T06:25:57.580ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1a7df7b-a3f1-4a03-9584-c2206daf2480", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94272025-04-30T06:25:57.580ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
94282025-04-30T06:25:57.592ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c839e73-b7ab-4f48-a054-d1238389b892", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94292025-04-30T06:25:57.593ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
94302025-04-30T06:25:57.596ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "966263a3-0bb4-4cd1-a482-619ef603f43d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94312025-04-30T06:25:57.596ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
94322025-04-30T06:25:57.597ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bccdf950-f86e-4fce-bba9-a94be617e16a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94332025-04-30T06:25:57.597ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
94342025-04-30T06:25:57.600ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3c37c76-e3f6-4678-8712-492d6e28347e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94352025-04-30T06:25:57.600ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
94362025-04-30T06:25:57.602ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb4cf242-c219-4020-8c73-4017d0739c2d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94372025-04-30T06:25:57.602ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
94382025-04-30T06:25:57.607ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94392025-04-30T06:25:57.607ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94402025-04-30T06:25:57.607ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b1d9757-7cee-47c8-926d-161117b38370", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94412025-04-30T06:25:57.607ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
94422025-04-30T06:25:57.608ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
94432025-04-30T06:25:57.610ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
94442025-04-30T06:25:57.617ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
94452025-04-30T06:25:57.618ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
94462025-04-30T06:25:57.623ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
94472025-04-30T06:25:57.623ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
94482025-04-30T06:25:57.625ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cab376d-54dc-49bd-a55d-2c13708966cf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94492025-04-30T06:25:57.625ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
94502025-04-30T06:25:57.629ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05f3cc1f-6801-4132-9188-8866ef016ddc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94512025-04-30T06:25:57.629ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
94522025-04-30T06:25:57.632ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
94532025-04-30T06:25:57.634ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b37354f-ac2f-4e48-b02e-faf81b636b71", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94542025-04-30T06:25:57.634ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
94552025-04-30T06:25:57.635ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35170ede-c1a9-47bc-a6de-2556b949e67b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94562025-04-30T06:25:57.635ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
94572025-04-30T06:25:57.637ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9db1e599-9982-43e7-8c8d-40b698362db5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94582025-04-30T06:25:57.637ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
94592025-04-30T06:25:57.638ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebcfb884-b3f6-4925-be11-9ce3af01836a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94602025-04-30T06:25:57.639ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
94612025-04-30T06:25:57.640ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e573c6a-e6cd-488d-ab13-258ee6cc1530", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94622025-04-30T06:25:57.641ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
94632025-04-30T06:25:57.641ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
94642025-04-30T06:25:57.644ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "54097d0e-7db8-4cb0-a726-1583b1e0313c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94652025-04-30T06:25:57.644ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
94662025-04-30T06:25:57.645ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c98b7224-7457-4be3-bf02-286411074bf5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94672025-04-30T06:25:57.645ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
94682025-04-30T06:25:57.647ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7415579b-d45a-4ea8-873d-fae0f1f5f53c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94692025-04-30T06:25:57.647ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
94702025-04-30T06:25:57.649ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6935cd3d-0aa5-4fc5-ab2b-819cbc67cd97", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94712025-04-30T06:25:57.649ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
94722025-04-30T06:25:57.651ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43262b11-7301-4027-848c-6e0cefdb897d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
94732025-04-30T06:25:57.651ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94742025-04-30T06:25:57.651ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94752025-04-30T06:25:57.651ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94762025-04-30T06:25:57.651ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94772025-04-30T06:25:57.652ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94782025-04-30T06:25:57.666ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94792025-04-30T06:25:57.672ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94802025-04-30T06:25:57.672ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
94812025-04-30T06:25:57.672ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94822025-04-30T06:25:57.672ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94832025-04-30T06:25:57.672ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94842025-04-30T06:25:57.673ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94852025-04-30T06:25:57.679ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94862025-04-30T06:25:57.689ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
94872025-04-30T06:25:57.689ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
94882025-04-30T06:25:57.689ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
94892025-04-30T06:25:57.690ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
94902025-04-30T06:25:57.691ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94912025-04-30T06:25:57.691ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94922025-04-30T06:25:57.691ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94932025-04-30T06:25:57.691ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
94942025-04-30T06:25:57.691ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94952025-04-30T06:25:57.691ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
94962025-04-30T06:25:57.694ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 42 56 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 BV..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:61060 req_id = 287beab1-51b1-4105-acef-76631685e31d
94972025-04-30T06:25:57.695ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16982,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61060 req_id = 287beab1-51b1-4105-acef-76631685e31d
94982025-04-30T06:25:57.695ZDEBGtest_replacement_sanity (store): query key key = _dendrite._tcp
94992025-04-30T06:25:57.695ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
95002025-04-30T06:25:57.698ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95012025-04-30T06:25:57.699ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61060 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 287beab1-51b1-4105-acef-76631685e31d
95022025-04-30T06:25:57.775ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
95032025-04-30T06:25:57.776ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
95042025-04-30T06:25:57.778ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:38810 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
95052025-04-30T06:25:57.778ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3222, tv_nsec: 577514508 } })
95062025-04-30T06:25:57.778ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
95072025-04-30T06:25:57.779ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
95082025-04-30T06:25:57.779ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
95092025-04-30T06:25:57.780ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95102025-04-30T06:25:57.786ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95112025-04-30T06:25:57.786ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95122025-04-30T06:25:57.787ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
95132025-04-30T06:25:57.787ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95142025-04-30T06:25:57.787ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:38810 req_id = 851d2244-b926-4f53-a9f2-138e8039f827 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
95152025-04-30T06:25:57.788ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 831 local_addr = [::1]:42668 method = GET remote_addr = [::1]:38810 req_id = 851d2244-b926-4f53-a9f2-138e8039f827 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
95162025-04-30T06:25:57.789ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a6 d6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51086 req_id = 0380ded2-b276-4d46-b1de-fec690a03999
95172025-04-30T06:25:57.789ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42710,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51086 req_id = 0380ded2-b276-4d46-b1de-fec690a03999
95182025-04-30T06:25:57.789ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
95192025-04-30T06:25:57.790ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51086 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0380ded2-b276-4d46-b1de-fec690a03999
95202025-04-30T06:25:57.790ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b1 4f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .O..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49680 req_id = 2e6e4a2b-04b3-43e4-bbdc-c68ac3e48ec3
95212025-04-30T06:25:57.790ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45391,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49680 req_id = 2e6e4a2b-04b3-43e4-bbdc-c68ac3e48ec3
95222025-04-30T06:25:57.790ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95232025-04-30T06:25:57.791ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49680 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2e6e4a2b-04b3-43e4-bbdc-c68ac3e48ec3
95242025-04-30T06:25:57.791ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "851d2244-b926-4f53-a9f2-138e8039f827", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
95252025-04-30T06:25:57.841ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
95262025-04-30T06:25:57.842ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
95272025-04-30T06:25:57.843ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:62514 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
95282025-04-30T06:25:57.844ZINFOtest_replacement_sanity (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
95292025-04-30T06:25:57.908ZINFOtest_replacement_sanity (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
95302025-04-30T06:25:57.908ZDEBGtest_replacement_sanity (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
95312025-04-30T06:25:57.908ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
95322025-04-30T06:25:57.909ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95332025-04-30T06:25:57.909ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
95342025-04-30T06:25:57.909ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95352025-04-30T06:25:57.910ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:62514 req_id = 4ce923d6-f199-44a4-8e0e-046c07fa1e13 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
95362025-04-30T06:25:57.911ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 795 local_addr = [::1]:47918 method = GET remote_addr = [::1]:62514 req_id = 4ce923d6-f199-44a4-8e0e-046c07fa1e13 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
95372025-04-30T06:25:57.912ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 35 e4 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 5............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64283 req_id = cbac8379-f6eb-4f73-9537-dd8c5ef88c8e
95382025-04-30T06:25:57.912ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13796,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64283 req_id = cbac8379-f6eb-4f73-9537-dd8c5ef88c8e
95392025-04-30T06:25:57.912ZDEBGtest_replacement_sanity (store): query key key = _dendrite._tcp
95402025-04-30T06:25:57.913ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
95412025-04-30T06:25:57.913ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95422025-04-30T06:25:57.913ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64283 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cbac8379-f6eb-4f73-9537-dd8c5ef88c8e
95432025-04-30T06:25:57.916ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ce923d6-f199-44a4-8e0e-046c07fa1e13", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
95442025-04-30T06:25:58.003ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
95452025-04-30T06:25:58.003ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
95462025-04-30T06:25:58.004ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:50272 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
95472025-04-30T06:25:58.004ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3222, tv_nsec: 803454388 } })
95482025-04-30T06:25:58.004ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
95492025-04-30T06:25:58.004ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
95502025-04-30T06:25:58.005ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:50272 req_id = fd870323-17de-404e-8ce8-8cecee87d73a sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
95512025-04-30T06:25:58.006ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 760 local_addr = [::1]:35744 method = GET remote_addr = [::1]:50272 req_id = fd870323-17de-404e-8ce8-8cecee87d73a response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
95522025-04-30T06:25:58.007ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f0 66 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .f..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49855 req_id = 0da22029-258a-4afc-943e-33246c0c35a8
95532025-04-30T06:25:58.007ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61542,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49855 req_id = 0da22029-258a-4afc-943e-33246c0c35a8
95542025-04-30T06:25:58.007ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
95552025-04-30T06:25:58.007ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49855 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0da22029-258a-4afc-943e-33246c0c35a8
95562025-04-30T06:25:58.008ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8f 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .t..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64359 req_id = 0c6f44d2-c54a-44ee-a9c8-87485a4d1a64
95572025-04-30T06:25:58.008ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36724,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64359 req_id = 0c6f44d2-c54a-44ee-a9c8-87485a4d1a64
95582025-04-30T06:25:58.008ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
95592025-04-30T06:25:58.009ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64359 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0c6f44d2-c54a-44ee-a9c8-87485a4d1a64
95602025-04-30T06:25:58.009ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd870323-17de-404e-8ce8-8cecee87d73a", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
95612025-04-30T06:25:58.056ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
95622025-04-30T06:25:58.056ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
95632025-04-30T06:25:58.057ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:44314 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
95642025-04-30T06:25:58.058ZINFOtest_replacement_sanity (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
95652025-04-30T06:25:58.119ZINFOtest_replacement_sanity (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
95662025-04-30T06:25:58.120ZDEBGtest_replacement_sanity (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
95672025-04-30T06:25:58.129ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:44314 req_id = dbb2c9ec-d589-43de-94ce-c086aae8b031 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95682025-04-30T06:25:58.130ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 730 local_addr = [::1]:35343 method = GET remote_addr = [::1]:44314 req_id = dbb2c9ec-d589-43de-94ce-c086aae8b031 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95692025-04-30T06:25:58.131ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbb2c9ec-d589-43de-94ce-c086aae8b031", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
95702025-04-30T06:25:58.134ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95712025-04-30T06:25:58.134ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95722025-04-30T06:25:58.135ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
95732025-04-30T06:25:58.141ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95742025-04-30T06:25:58.141ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95752025-04-30T06:25:58.141ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95762025-04-30T06:25:58.141ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95772025-04-30T06:25:58.141ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95782025-04-30T06:25:58.149ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95792025-04-30T06:25:58.158ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95802025-04-30T06:25:58.158ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95812025-04-30T06:25:58.158ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95822025-04-30T06:25:58.158ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95832025-04-30T06:25:58.193ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
95842025-04-30T06:25:58.203ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
95852025-04-30T06:25:58.204ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
95862025-04-30T06:25:58.204ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 952.080929ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
95872025-04-30T06:25:58.204ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
95882025-04-30T06:25:58.204ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95892025-04-30T06:25:58.204ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95902025-04-30T06:25:58.204ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95912025-04-30T06:25:58.204ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95922025-04-30T06:25:58.205ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95932025-04-30T06:25:58.207ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95942025-04-30T06:25:58.212ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95952025-04-30T06:25:58.212ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95962025-04-30T06:25:58.212ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95972025-04-30T06:25:58.212ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95982025-04-30T06:25:58.217ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
95992025-04-30T06:25:58.217ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
96002025-04-30T06:25:58.218ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b4 e6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57457 req_id = 275e748d-3953-4308-b4c8-eccb8b57bfa7
96012025-04-30T06:25:58.218ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46310,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57457 req_id = 275e748d-3953-4308-b4c8-eccb8b57bfa7
96022025-04-30T06:25:58.218ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
96032025-04-30T06:25:58.218ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
96042025-04-30T06:25:58.218ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96052025-04-30T06:25:58.219ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57457 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 275e748d-3953-4308-b4c8-eccb8b57bfa7
96062025-04-30T06:25:58.219ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3223, tv_nsec: 18594871 } })
96072025-04-30T06:25:58.219ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
96082025-04-30T06:25:58.219ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
96092025-04-30T06:25:58.220ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 7e 72 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ~r..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61725 req_id = ceebcfb9-c941-4c44-89e9-b1a34228ca43
96102025-04-30T06:25:58.220ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32370,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61725 req_id = ceebcfb9-c941-4c44-89e9-b1a34228ca43
96112025-04-30T06:25:58.221ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
96122025-04-30T06:25:58.221ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61725 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ceebcfb9-c941-4c44-89e9-b1a34228ca43
96132025-04-30T06:25:58.221ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 16 3b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .;..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56931 req_id = 092002ee-655f-49ef-a9b3-c5970dd5cdf7
96142025-04-30T06:25:58.221ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5691,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56931 req_id = 092002ee-655f-49ef-a9b3-c5970dd5cdf7
96152025-04-30T06:25:58.221ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96162025-04-30T06:25:58.221ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56931 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 092002ee-655f-49ef-a9b3-c5970dd5cdf7
96172025-04-30T06:25:58.358ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
96182025-04-30T06:25:58.359ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96192025-04-30T06:25:58.359ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96202025-04-30T06:25:58.359ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96212025-04-30T06:25:58.359ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
96222025-04-30T06:25:58.361ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96232025-04-30T06:25:58.361ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 27 e4 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c '............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61737 req_id = 98d446c9-f32d-48d3-943a-439074021201
96242025-04-30T06:25:58.361ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10212,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61737 req_id = 98d446c9-f32d-48d3-943a-439074021201
96252025-04-30T06:25:58.362ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
96262025-04-30T06:25:58.362ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10212, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61737 req_id = 98d446c9-f32d-48d3-943a-439074021201
96272025-04-30T06:25:58.362ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
96282025-04-30T06:25:58.362ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
96292025-04-30T06:25:58.362ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
96302025-04-30T06:25:58.364ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96312025-04-30T06:25:58.368ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96322025-04-30T06:25:58.368ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96332025-04-30T06:25:58.369ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
96342025-04-30T06:25:58.369ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96352025-04-30T06:25:58.373ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af7bd028-b6d5-4009-933f-238df735e367", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96362025-04-30T06:25:58.374ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
96372025-04-30T06:25:58.377ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96382025-04-30T06:25:58.378ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
96392025-04-30T06:25:58.380ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96402025-04-30T06:25:58.383ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bdace85-363a-403b-aaf4-48f9eea85f09", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96412025-04-30T06:25:58.383ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
96422025-04-30T06:25:58.390ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca19a7f1-c320-44bd-82fd-11d24b5cbc7c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96432025-04-30T06:25:58.391ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
96442025-04-30T06:25:58.395ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "47c2152e-2fe7-4e9f-a38f-be080cff6b57", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96452025-04-30T06:25:58.395ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
96462025-04-30T06:25:58.398ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7bf2890a-2c33-48ca-8840-0ec0c65e0fea", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96472025-04-30T06:25:58.399ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
96482025-04-30T06:25:58.402ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0fe6609-550f-4c49-901f-dc1f3ad3a3d7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96492025-04-30T06:25:58.402ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
96502025-04-30T06:25:58.405ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74929b63-2b5f-4305-8de7-b3cbcdb384a5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96512025-04-30T06:25:58.405ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
96522025-04-30T06:25:58.408ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96532025-04-30T06:25:58.408ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96542025-04-30T06:25:58.413ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
96552025-04-30T06:25:58.414ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96562025-04-30T06:25:58.414ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96572025-04-30T06:25:58.414ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96582025-04-30T06:25:58.414ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
96592025-04-30T06:25:58.415ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1357a77-97d1-41a1-91ba-deb292db5ac3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96602025-04-30T06:25:58.416ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
96612025-04-30T06:25:58.416ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3384380-c815-46d3-af6e-e72455811f2b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96622025-04-30T06:25:58.417ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
96632025-04-30T06:25:58.418ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1390a9aa-d044-4514-910d-efe4b6c88fa3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96642025-04-30T06:25:58.418ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
96652025-04-30T06:25:58.419ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7ed09b8-367b-4489-9e44-4067a2e4e856", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96662025-04-30T06:25:58.419ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
96672025-04-30T06:25:58.421ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ad63acf-6ace-4b1e-b1f8-11b84a944e87", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96682025-04-30T06:25:58.421ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
96692025-04-30T06:25:58.422ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a05c2d-f8d4-4161-aa36-a0310fa2fd7d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96702025-04-30T06:25:58.422ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
96712025-04-30T06:25:58.425ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29d342a7-60a0-4614-a0a3-e7405d8b66a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96722025-04-30T06:25:58.425ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
96732025-04-30T06:25:58.427ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e776295-7eb8-43f5-9c32-3dd6a29abb95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96742025-04-30T06:25:58.427ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
96752025-04-30T06:25:58.470ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96762025-04-30T06:25:58.471ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96772025-04-30T06:25:58.471ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96782025-04-30T06:25:58.471ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
96792025-04-30T06:25:58.472ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96802025-04-30T06:25:58.473ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a258be8-ea4c-4f24-b9ca-84060a84c8a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96812025-04-30T06:25:58.473ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
96822025-04-30T06:25:58.477ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96832025-04-30T06:25:58.485ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96842025-04-30T06:25:58.485ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96852025-04-30T06:25:58.485ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
96862025-04-30T06:25:58.487ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96872025-04-30T06:25:58.491ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d757c23d-658c-4c25-b170-e604bffa9cc3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96882025-04-30T06:25:58.491ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
96892025-04-30T06:25:58.491ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96902025-04-30T06:25:58.491ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
96912025-04-30T06:25:58.492ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96922025-04-30T06:25:58.494ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d178816d-4a60-45fb-a9a5-606a5e959b22", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96932025-04-30T06:25:58.494ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
96942025-04-30T06:25:58.497ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b5b35fb-20f6-40d0-8f4c-1395c2a4d1a4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96952025-04-30T06:25:58.497ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
96962025-04-30T06:25:58.498ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af036591-e407-4dda-86e3-eb2a83b97d7d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96972025-04-30T06:25:58.499ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
96982025-04-30T06:25:58.501ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c915a7e-378d-4bfe-9c2e-7e71cb4780e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
96992025-04-30T06:25:58.501ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
97002025-04-30T06:25:58.502ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5e4c6c1-dddb-4b9c-8826-93ac49d22606", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
97012025-04-30T06:25:58.502ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
97022025-04-30T06:25:58.503ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a531dd5-bd28-4c24-9ca8-14d988de0596", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
97032025-04-30T06:25:58.504ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
97042025-04-30T06:25:58.506ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "11f1259a-e4fe-4edb-a63c-62d564d59935", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
97052025-04-30T06:25:58.507ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
97062025-04-30T06:25:58.510ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a85e75-1d5c-4aae-9538-eb4031509411", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
97072025-04-30T06:25:58.510ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
97082025-04-30T06:25:58.513ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e96d6ba7-65bf-4d42-84a2-5e21f689f826", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
97092025-04-30T06:25:58.515ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
97102025-04-30T06:25:58.523ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bcbb2a8-bf3e-4d57-a5ee-2721123d7c00", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:57 GMT"} })
97112025-04-30T06:25:58.523ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
97122025-04-30T06:25:58.524ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
97132025-04-30T06:25:58.528ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f1b3cc-9d2d-4298-861f-a3ebde07bd81", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97142025-04-30T06:25:58.528ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
97152025-04-30T06:25:58.532ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62f28ded-19e4-4240-80df-27a172fff7d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97162025-04-30T06:25:58.532ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
97172025-04-30T06:25:58.535ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "642def2b-dd6b-4e46-8063-b0af7c0cd158", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97182025-04-30T06:25:58.536ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
97192025-04-30T06:25:58.537ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df9a0ee8-3648-40ea-a13f-6dee36af7bcd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97202025-04-30T06:25:58.537ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
97212025-04-30T06:25:58.538ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b81d0a1c-880f-475a-b913-c05c6229f49b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97222025-04-30T06:25:58.538ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
97232025-04-30T06:25:58.539ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "de2fbb97-7f65-47af-8114-27a5777d8dfa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97242025-04-30T06:25:58.539ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
97252025-04-30T06:25:58.541ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c5f2ba-3ae5-4812-918f-e3b5491ec969", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97262025-04-30T06:25:58.541ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
97272025-04-30T06:25:58.541ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b38833c-694c-4375-bce4-66342d216178", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97282025-04-30T06:25:58.542ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
97292025-04-30T06:25:58.544ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7d2828e-fd7f-4ef0-a933-8b635a1de080", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97302025-04-30T06:25:58.545ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
97312025-04-30T06:25:58.548ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39af11b5-ebfa-4b93-b2a8-569fa7d26377", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97322025-04-30T06:25:58.548ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
97332025-04-30T06:25:58.552ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d41b9b75-228a-48b9-8128-8d5ba6853044", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97342025-04-30T06:25:58.552ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
97352025-04-30T06:25:58.554ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d037a6d-6d95-4186-9a29-f7fccdfef14f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97362025-04-30T06:25:58.554ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
97372025-04-30T06:25:58.556ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76757bf0-8cd3-4f43-b7f6-e6072b54cb0d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97382025-04-30T06:25:58.557ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
97392025-04-30T06:25:58.560ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df8f71e9-e68a-4e89-88c1-a780848f9f9e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97402025-04-30T06:25:58.560ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
97412025-04-30T06:25:58.561ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7b555d8-f8d3-4ff0-8cd3-081b23952b78", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97422025-04-30T06:25:58.561ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
97432025-04-30T06:25:58.562ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbf4803a-ca61-4a9d-a44b-4f470f19381d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97442025-04-30T06:25:58.562ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
97452025-04-30T06:25:58.563ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a292360-cf3f-43f0-840f-38b9069c6b2a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97462025-04-30T06:25:58.563ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
97472025-04-30T06:25:58.564ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "239f6387-3bbb-406b-b362-c8d5c05cb7af", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97482025-04-30T06:25:58.564ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
97492025-04-30T06:25:58.564ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
97502025-04-30T06:25:58.569ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "da12a382-d7ea-4bf4-91e2-72f22d163ecc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97512025-04-30T06:25:58.569ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
97522025-04-30T06:25:58.572ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48bdd1b2-3631-459c-a778-80c98e8e2f8b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97532025-04-30T06:25:58.572ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
97542025-04-30T06:25:58.575ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97552025-04-30T06:25:58.575ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97562025-04-30T06:25:58.575ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97572025-04-30T06:25:58.575ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
97582025-04-30T06:25:58.577ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97592025-04-30T06:25:58.577ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db552bf-1f8d-4194-92b3-ae8984856caf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97602025-04-30T06:25:58.577ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
97612025-04-30T06:25:58.581ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a42d065a-71e1-4327-884a-e68e6f2b1cbd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97622025-04-30T06:25:58.581ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
97632025-04-30T06:25:58.582ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97642025-04-30T06:25:58.590ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97652025-04-30T06:25:58.590ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97662025-04-30T06:25:58.590ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
97672025-04-30T06:25:58.590ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97682025-04-30T06:25:58.594ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b169477e-0367-4ce3-8ebe-75c77ff5f70e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
97692025-04-30T06:25:58.594ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97702025-04-30T06:25:58.594ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97712025-04-30T06:25:58.595ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97722025-04-30T06:25:58.595ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97732025-04-30T06:25:58.595ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97742025-04-30T06:25:58.596ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97752025-04-30T06:25:58.596ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
97762025-04-30T06:25:58.597ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97772025-04-30T06:25:58.599ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97782025-04-30T06:25:58.604ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97792025-04-30T06:25:58.605ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97802025-04-30T06:25:58.605ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97812025-04-30T06:25:58.605ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97822025-04-30T06:25:58.605ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97832025-04-30T06:25:58.605ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97842025-04-30T06:25:58.607ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97852025-04-30T06:25:58.607ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
97862025-04-30T06:25:58.608ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
97872025-04-30T06:25:58.612ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
97882025-04-30T06:25:58.613ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97892025-04-30T06:25:58.619ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97902025-04-30T06:25:58.619ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97912025-04-30T06:25:58.619ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97922025-04-30T06:25:58.620ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97932025-04-30T06:25:58.629ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
97942025-04-30T06:25:58.629ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
97952025-04-30T06:25:58.683ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
97962025-04-30T06:25:58.683ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
97972025-04-30T06:25:58.684ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:46821 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
97982025-04-30T06:25:58.686ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:46821 req_id = caa0eafc-faf0-4cc9-ab54-20d45ff546d6 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
97992025-04-30T06:25:58.686ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 479 local_addr = [::1]:42668 method = GET remote_addr = [::1]:46821 req_id = caa0eafc-faf0-4cc9-ab54-20d45ff546d6 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
98002025-04-30T06:25:58.687ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "caa0eafc-faf0-4cc9-ab54-20d45ff546d6", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98012025-04-30T06:25:58.741ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
98022025-04-30T06:25:58.741ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
98032025-04-30T06:25:58.742ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:50160 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
98042025-04-30T06:25:58.743ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
98052025-04-30T06:25:58.743ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
98062025-04-30T06:25:58.744ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
98072025-04-30T06:25:58.744ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:50160 req_id = f89cbde3-ce1f-4e91-a69f-d5591db624fe sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
98082025-04-30T06:25:58.744ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 418 local_addr = [::1]:47918 method = GET remote_addr = [::1]:50160 req_id = f89cbde3-ce1f-4e91-a69f-d5591db624fe response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
98092025-04-30T06:25:58.745ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f89cbde3-ce1f-4e91-a69f-d5591db624fe", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98102025-04-30T06:25:58.844ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
98112025-04-30T06:25:58.844ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
98122025-04-30T06:25:58.845ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:52353 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
98132025-04-30T06:25:58.847ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:52353 req_id = 496f2041-d0aa-49d4-90c4-6075ef807c6b sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
98142025-04-30T06:25:58.848ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 331 local_addr = [::1]:35744 method = GET remote_addr = [::1]:52353 req_id = 496f2041-d0aa-49d4-90c4-6075ef807c6b response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
98152025-04-30T06:25:58.848ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "496f2041-d0aa-49d4-90c4-6075ef807c6b", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98162025-04-30T06:25:58.919ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
98172025-04-30T06:25:58.920ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
98182025-04-30T06:25:58.920ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:33970 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
98192025-04-30T06:25:58.921ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:33970 req_id = f1129d8b-7ec4-493c-a3f9-8e53dcf545b2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98202025-04-30T06:25:58.922ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 544 local_addr = [::1]:35343 method = GET remote_addr = [::1]:33970 req_id = f1129d8b-7ec4-493c-a3f9-8e53dcf545b2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
98212025-04-30T06:25:58.922ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1129d8b-7ec4-493c-a3f9-8e53dcf545b2", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98222025-04-30T06:25:58.925ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98232025-04-30T06:25:58.925ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
98242025-04-30T06:25:58.925ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
98252025-04-30T06:25:58.930ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98262025-04-30T06:25:58.930ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98272025-04-30T06:25:58.930ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98282025-04-30T06:25:58.930ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98292025-04-30T06:25:58.930ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98302025-04-30T06:25:58.933ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98312025-04-30T06:25:58.938ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98322025-04-30T06:25:58.938ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98332025-04-30T06:25:58.938ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98342025-04-30T06:25:58.939ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98352025-04-30T06:25:58.952ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
98362025-04-30T06:25:58.985ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
98372025-04-30T06:25:58.985ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
98382025-04-30T06:25:58.985ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 781.769564ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
98392025-04-30T06:25:58.985ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
98402025-04-30T06:25:58.986ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
98412025-04-30T06:25:58.986ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98422025-04-30T06:25:58.986ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98432025-04-30T06:25:58.986ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98442025-04-30T06:25:58.986ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98452025-04-30T06:25:58.993ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98462025-04-30T06:25:58.998ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98472025-04-30T06:25:58.998ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98482025-04-30T06:25:58.998ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98492025-04-30T06:25:58.999ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98502025-04-30T06:25:59.004ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98512025-04-30T06:25:59.004ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98522025-04-30T06:25:59.004ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98532025-04-30T06:25:59.004ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
98542025-04-30T06:25:59.005ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98552025-04-30T06:25:59.005ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
98562025-04-30T06:25:59.005ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
98572025-04-30T06:25:59.008ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98582025-04-30T06:25:59.015ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98592025-04-30T06:25:59.015ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98602025-04-30T06:25:59.015ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
98612025-04-30T06:25:59.015ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98622025-04-30T06:25:59.016ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4d 3b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 M;..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54957 req_id = a3892c62-8476-4b92-9820-d7373d7fa20b
98632025-04-30T06:25:59.016ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19771,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54957 req_id = a3892c62-8476-4b92-9820-d7373d7fa20b
98642025-04-30T06:25:59.016ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
98652025-04-30T06:25:59.017ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
98662025-04-30T06:25:59.018ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98672025-04-30T06:25:59.018ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54957 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a3892c62-8476-4b92-9820-d7373d7fa20b
98682025-04-30T06:25:59.019ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3223, tv_nsec: 818115744 } })
98692025-04-30T06:25:59.019ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
98702025-04-30T06:25:59.019ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
98712025-04-30T06:25:59.020ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d6 01 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57301 req_id = 9650b1ab-f340-4c1b-b43c-44dc343907fb
98722025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54785,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57301 req_id = 9650b1ab-f340-4c1b-b43c-44dc343907fb
98732025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
98742025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57301 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9650b1ab-f340-4c1b-b43c-44dc343907fb
98752025-04-30T06:25:59.020ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 09 39 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .9..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52476 req_id = 690ad6e6-375f-41be-aade-725d90ae81d0
98762025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2361,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52476 req_id = 690ad6e6-375f-41be-aade-725d90ae81d0
98772025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98782025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52476 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 690ad6e6-375f-41be-aade-725d90ae81d0
98792025-04-30T06:25:59.020ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98802025-04-30T06:25:59.020ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
98812025-04-30T06:25:59.021ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98822025-04-30T06:25:59.134ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
98832025-04-30T06:25:59.136ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 94 e5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65184 req_id = c3537a62-860f-4e6b-a883-8aeef9119c19
98842025-04-30T06:25:59.136ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38117,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65184 req_id = c3537a62-860f-4e6b-a883-8aeef9119c19
98852025-04-30T06:25:59.136ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
98862025-04-30T06:25:59.137ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 38117, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65184 req_id = c3537a62-860f-4e6b-a883-8aeef9119c19
98872025-04-30T06:25:59.137ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
98882025-04-30T06:25:59.137ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
98892025-04-30T06:25:59.137ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
98902025-04-30T06:25:59.140ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4a30cca-b5f6-4672-9972-ea5d0a557293", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98912025-04-30T06:25:59.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
98922025-04-30T06:25:59.145ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f559cd5-cf74-4e98-bbb8-b7c47977d6b9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98932025-04-30T06:25:59.145ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
98942025-04-30T06:25:59.194ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "939c2fa2-49d2-4813-8f3d-b18b47b61983", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98952025-04-30T06:25:59.195ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
98962025-04-30T06:25:59.207ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb0cd8d3-8976-4e66-a979-c821c376f325", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
98972025-04-30T06:25:59.207ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
98982025-04-30T06:25:59.211ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
98992025-04-30T06:25:59.213ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba944437-782e-43b1-b68a-89d36923d735", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99002025-04-30T06:25:59.214ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
99012025-04-30T06:25:59.218ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d4d82a8-a959-44c7-9fc4-d8e0ec889cf4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99022025-04-30T06:25:59.218ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
99032025-04-30T06:25:59.221ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee7a1ba-1e30-4ca3-b86f-548857085ec9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99042025-04-30T06:25:59.221ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
99052025-04-30T06:25:59.225ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c7fa2b1-abda-41ec-9bd7-655ac2037837", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99062025-04-30T06:25:59.225ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
99072025-04-30T06:25:59.226ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3b2b98c-0219-45b5-b3a5-4874e95f38cc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99082025-04-30T06:25:59.226ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
99092025-04-30T06:25:59.228ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1948b7e2-36c7-463b-930c-7c0e767978ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99102025-04-30T06:25:59.228ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
99112025-04-30T06:25:59.233ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ad1e948-49bc-4af7-945a-c9bbd3b3707e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99122025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
99132025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c338568-4efa-4eb4-bd88-5e004ad7fa0f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99142025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
99152025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62cd8fa3-e1ec-4035-b3de-b4f9c75ae250", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99162025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
99172025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6298e95-5ac3-48f5-b460-78c83a3db455", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99182025-04-30T06:25:59.234ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
99192025-04-30T06:25:59.237ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "41608349-66ad-4155-b4b8-37dd0029f90c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99202025-04-30T06:25:59.237ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
99212025-04-30T06:25:59.240ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0653676-44f6-4634-87c9-93c0934edea1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99222025-04-30T06:25:59.240ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
99232025-04-30T06:25:59.244ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "521cd8cf-2c7f-414e-a0e1-77ad2b3652eb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99242025-04-30T06:25:59.245ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
99252025-04-30T06:25:59.245ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b64170e2-0509-453a-a6e8-599d168f8b16", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99262025-04-30T06:25:59.245ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
99272025-04-30T06:25:59.247ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "593fb2a7-c691-4570-ad92-fd3958ab6a30", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99282025-04-30T06:25:59.247ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
99292025-04-30T06:25:59.249ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc4df4ac-c8ba-4834-822a-fc9444a76a29", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99302025-04-30T06:25:59.250ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
99312025-04-30T06:25:59.251ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94279aac-b380-4015-b320-4c6e1a391d3b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99322025-04-30T06:25:59.251ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
99332025-04-30T06:25:59.252ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56564f25-7d30-4e81-aabb-1dc5f22cb836", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99342025-04-30T06:25:59.252ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
99352025-04-30T06:25:59.256ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ec9ed39-fd01-4c4a-8778-9f83efb573b5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99362025-04-30T06:25:59.256ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
99372025-04-30T06:25:59.262ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99382025-04-30T06:25:59.263ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99392025-04-30T06:25:59.263ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99402025-04-30T06:25:59.263ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
99412025-04-30T06:25:59.265ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99422025-04-30T06:25:59.265ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc4734bd-fda6-4687-ae67-b30fb3e60d07", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99432025-04-30T06:25:59.265ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
99442025-04-30T06:25:59.268ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99452025-04-30T06:25:59.278ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99462025-04-30T06:25:59.278ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99472025-04-30T06:25:59.278ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
99482025-04-30T06:25:59.279ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99492025-04-30T06:25:59.281ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3d98da3-99d6-4a51-8621-38582c19add5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99502025-04-30T06:25:59.281ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
99512025-04-30T06:25:59.285ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99522025-04-30T06:25:59.285ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
99532025-04-30T06:25:59.290ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99542025-04-30T06:25:59.290ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af9db293-5499-4ff6-8cf0-14211308ebb6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99552025-04-30T06:25:59.291ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
99562025-04-30T06:25:59.295ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c765d020-c14a-49bf-92b4-0adfdea1f125", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99572025-04-30T06:25:59.295ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
99582025-04-30T06:25:59.301ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
99592025-04-30T06:25:59.301ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99602025-04-30T06:25:59.302ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 06dcbc9b-fa37-4f78-a125-4c994ee4b7cd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99612025-04-30T06:25:59.304ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 06dcbc9b-fa37-4f78-a125-4c994ee4b7cd resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99622025-04-30T06:25:59.305ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 06dcbc9b-fa37-4f78-a125-4c994ee4b7cd roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99632025-04-30T06:25:59.308ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 06dcbc9b-fa37-4f78-a125-4c994ee4b7cd resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99642025-04-30T06:25:59.311ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 8286 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 06dcbc9b-fa37-4f78-a125-4c994ee4b7cd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99652025-04-30T06:25:59.312ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "06dcbc9b-fa37-4f78-a125-4c994ee4b7cd", "content-length": "405", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99662025-04-30T06:25:59.312ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
99672025-04-30T06:25:59.312ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 901ee903-3911-4ba4-b330-495b107ea061 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
99682025-04-30T06:25:59.313ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 901ee903-3911-4ba4-b330-495b107ea061 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
99692025-04-30T06:25:59.313ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 901ee903-3911-4ba4-b330-495b107ea061 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
99702025-04-30T06:25:59.323ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 901ee903-3911-4ba4-b330-495b107ea061 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
99712025-04-30T06:25:59.325ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c3f24d5-4e77-4af0-a3df-a6f7c4b836b3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99722025-04-30T06:25:59.325ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
99732025-04-30T06:25:59.329ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 16526 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 901ee903-3911-4ba4-b330-495b107ea061 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
99742025-04-30T06:25:59.329ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "901ee903-3911-4ba4-b330-495b107ea061", "content-length": "29", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99752025-04-30T06:25:59.330ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99762025-04-30T06:25:59.330ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
99772025-04-30T06:25:59.330ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
99782025-04-30T06:25:59.339ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "35a9fae0-c5a7-4173-b3f2-f5cef915a12d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99792025-04-30T06:25:59.340ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
99802025-04-30T06:25:59.346ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d64a70c1-54c5-4ef7-adc2-b5acab867f3c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99812025-04-30T06:25:59.347ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
99822025-04-30T06:25:59.349ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddb1235d-551f-4413-9ebb-4b6530b948c9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99832025-04-30T06:25:59.349ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
99842025-04-30T06:25:59.351ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2c1a67-2eb4-4ec7-ab09-503a40d3933e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99852025-04-30T06:25:59.352ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
99862025-04-30T06:25:59.353ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
99872025-04-30T06:25:59.359ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e5c6ea5-2128-4755-a7db-7f778a20c12f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99882025-04-30T06:25:59.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
99892025-04-30T06:25:59.359ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db2c8c71-df5e-4efc-98b1-5bd42f3c2adc", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99902025-04-30T06:25:59.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
99912025-04-30T06:25:59.359ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01a787bb-2096-44fe-91a9-ce51c2696613", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99922025-04-30T06:25:59.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
99932025-04-30T06:25:59.362ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05a59ced-535f-40bd-87fd-87383ba100eb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99942025-04-30T06:25:59.363ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
99952025-04-30T06:25:59.367ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "df28345f-6a1b-4108-9d1d-7fd042b987e5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99962025-04-30T06:25:59.367ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
99972025-04-30T06:25:59.371ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e44b5814-c2bb-4692-bb78-7066f764cd4a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
99982025-04-30T06:25:59.371ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
99992025-04-30T06:25:59.375ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed6c2698-b524-418f-a0ed-d57c672428e4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100002025-04-30T06:25:59.376ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
100012025-04-30T06:25:59.380ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae3daf7e-3076-4a4b-92e8-e7d92419950a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100022025-04-30T06:25:59.381ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
100032025-04-30T06:25:59.384ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba90e45d-8a48-480f-9e5d-07ee19e71d7d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100042025-04-30T06:25:59.384ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
100052025-04-30T06:25:59.386ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26720e06-7039-40b5-8570-a033e756df55", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100062025-04-30T06:25:59.386ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
100072025-04-30T06:25:59.387ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8eda344-ecf4-40ee-ba51-2dc4442a7296", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100082025-04-30T06:25:59.388ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
100092025-04-30T06:25:59.389ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "214fd9ad-f405-4dbf-9851-ca02dc39b89f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100102025-04-30T06:25:59.389ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
100112025-04-30T06:25:59.390ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a35632fc-1c6a-48b6-9d2e-09d98094aee0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100122025-04-30T06:25:59.391ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
100132025-04-30T06:25:59.391ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
100142025-04-30T06:25:59.395ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "94385d5b-10a0-42a5-994f-9409a2862304", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100152025-04-30T06:25:59.395ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
100162025-04-30T06:25:59.397ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8031680e-09a1-4aad-8ba0-fc9bca42caa6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100172025-04-30T06:25:59.397ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
100182025-04-30T06:25:59.399ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3150399-1019-4b49-967d-9f08a6ea3a4c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100192025-04-30T06:25:59.400ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
100202025-04-30T06:25:59.402ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba2c82ce-748b-404d-afdb-c053ad05a6b0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100212025-04-30T06:25:59.402ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
100222025-04-30T06:25:59.403ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4634c90a-2c04-4cfc-bb3b-018c0feede35", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100232025-04-30T06:25:59.403ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100242025-04-30T06:25:59.404ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100252025-04-30T06:25:59.404ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100262025-04-30T06:25:59.404ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100272025-04-30T06:25:59.404ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100282025-04-30T06:25:59.405ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100292025-04-30T06:25:59.405ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100302025-04-30T06:25:59.405ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100312025-04-30T06:25:59.405ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
100322025-04-30T06:25:59.405ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100332025-04-30T06:25:59.407ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100342025-04-30T06:25:59.424ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100352025-04-30T06:25:59.424ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
100362025-04-30T06:25:59.424ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100372025-04-30T06:25:59.424ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100382025-04-30T06:25:59.427ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100392025-04-30T06:25:59.427ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100402025-04-30T06:25:59.427ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100412025-04-30T06:25:59.432ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
100422025-04-30T06:25:59.432ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100432025-04-30T06:25:59.432ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100442025-04-30T06:25:59.433ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100452025-04-30T06:25:59.434ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100462025-04-30T06:25:59.434ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100472025-04-30T06:25:59.437ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100482025-04-30T06:25:59.442ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100492025-04-30T06:25:59.442ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100502025-04-30T06:25:59.442ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
100512025-04-30T06:25:59.442ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100522025-04-30T06:25:59.443ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100532025-04-30T06:25:59.444ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100542025-04-30T06:25:59.444ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100552025-04-30T06:25:59.444ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100562025-04-30T06:25:59.503ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
100572025-04-30T06:25:59.503ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
100582025-04-30T06:25:59.506ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:46894 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
100592025-04-30T06:25:59.506ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
100602025-04-30T06:25:59.508ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
100612025-04-30T06:25:59.508ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100622025-04-30T06:25:59.508ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
100632025-04-30T06:25:59.509ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100642025-04-30T06:25:59.510ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:46894 req_id = fc50d6f9-0109-4fd2-b3eb-bc49d5c01c22 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
100652025-04-30T06:25:59.511ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = 127.0.0.1:50502 method = GET remote_addr = 127.0.0.1:65326 req_id = ef320b22-6885-4dbf-b38e-290114bc41c8 uri = /c31da0ae-930d-4dd9-a897-159d5a6cd420
100662025-04-30T06:25:59.523ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 13067 local_addr = [::1]:42668 method = GET remote_addr = [::1]:46894 req_id = fc50d6f9-0109-4fd2-b3eb-bc49d5c01c22 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
100672025-04-30T06:25:59.525ZINFOtest_replacement_sanity (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 13045 local_addr = 127.0.0.1:50502 method = GET remote_addr = 127.0.0.1:65326 req_id = ef320b22-6885-4dbf-b38e-290114bc41c8 response_code = 200 uri = /c31da0ae-930d-4dd9-a897-159d5a6cd420
100682025-04-30T06:25:59.528ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc50d6f9-0109-4fd2-b3eb-bc49d5c01c22", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:58 GMT"} })
100692025-04-30T06:25:59.622ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
100702025-04-30T06:25:59.622ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
100712025-04-30T06:25:59.622ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
100722025-04-30T06:25:59.633ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
100732025-04-30T06:25:59.633ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
100742025-04-30T06:25:59.633ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
100752025-04-30T06:25:59.637ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:38212 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
100762025-04-30T06:25:59.637ZDEBGtest_replacement_sanity (results-sink): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100772025-04-30T06:25:59.637ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 6 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100782025-04-30T06:25:59.654ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 6 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
100792025-04-30T06:25:59.660ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.fields_string FORMAT Native
100802025-04-30T06:25:59.662ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:38212 req_id = de0a80eb-6f2b-47a3-bbe1-4c6f9d52eeca sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
100812025-04-30T06:25:59.662ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 322 local_addr = [::1]:47918 method = GET remote_addr = [::1]:38212 req_id = de0a80eb-6f2b-47a3-bbe1-4c6f9d52eeca response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
100822025-04-30T06:25:59.673ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "de0a80eb-6f2b-47a3-bbe1-4c6f9d52eeca", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
100832025-04-30T06:25:59.754ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
100842025-04-30T06:25:59.754ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
100852025-04-30T06:25:59.756ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:36172 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
100862025-04-30T06:25:59.757ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 12 table_name = fields_string
100872025-04-30T06:25:59.757ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
100882025-04-30T06:25:59.757ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
100892025-04-30T06:25:59.758ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:36172 req_id = 93fdb85f-7d5e-484d-8027-5092d44387c7 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
100902025-04-30T06:25:59.759ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1642 local_addr = [::1]:35744 method = GET remote_addr = [::1]:36172 req_id = 93fdb85f-7d5e-484d-8027-5092d44387c7 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
100912025-04-30T06:25:59.761ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "93fdb85f-7d5e-484d-8027-5092d44387c7", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
100922025-04-30T06:25:59.812ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
100932025-04-30T06:25:59.812ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
100942025-04-30T06:25:59.813ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:40767 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
100952025-04-30T06:25:59.813ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 5 table_name = fields_u16
100962025-04-30T06:25:59.813ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 5 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
100972025-04-30T06:25:59.819ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:40767 req_id = d0bf6ec4-09fa-4f9a-acca-f487adaa6a7a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100982025-04-30T06:25:59.820ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1250 local_addr = [::1]:35343 method = GET remote_addr = [::1]:40767 req_id = d0bf6ec4-09fa-4f9a-acca-f487adaa6a7a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
100992025-04-30T06:25:59.822ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0bf6ec4-09fa-4f9a-acca-f487adaa6a7a", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101002025-04-30T06:25:59.825ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101012025-04-30T06:25:59.825ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
101022025-04-30T06:25:59.825ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
101032025-04-30T06:25:59.834ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101042025-04-30T06:25:59.839ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101052025-04-30T06:25:59.840ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101062025-04-30T06:25:59.840ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101072025-04-30T06:25:59.843ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101082025-04-30T06:25:59.847ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 5 table_name = fields_uuid
101092025-04-30T06:25:59.847ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 19 n_rows = 5 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
101102025-04-30T06:25:59.849ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101112025-04-30T06:25:59.853ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101122025-04-30T06:25:59.854ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101132025-04-30T06:25:59.854ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101142025-04-30T06:25:59.855ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101152025-04-30T06:25:59.869ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 5 table_name = measurements_histogramu64
101162025-04-30T06:25:59.869ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
101172025-04-30T06:25:59.882ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
101182025-04-30T06:25:59.882ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
101192025-04-30T06:25:59.884ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
101202025-04-30T06:25:59.914ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
101212025-04-30T06:25:59.915ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
101222025-04-30T06:25:59.915ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 929.483124ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
101232025-04-30T06:25:59.915ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
101242025-04-30T06:25:59.915ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
101252025-04-30T06:25:59.915ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101262025-04-30T06:25:59.915ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101272025-04-30T06:25:59.915ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101282025-04-30T06:25:59.925ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101292025-04-30T06:25:59.931ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101302025-04-30T06:25:59.951ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
101312025-04-30T06:25:59.951ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101322025-04-30T06:25:59.951ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101332025-04-30T06:25:59.952ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101342025-04-30T06:25:59.952ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101352025-04-30T06:25:59.952ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101362025-04-30T06:25:59.952ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101372025-04-30T06:25:59.952ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
101382025-04-30T06:25:59.953ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101392025-04-30T06:25:59.956ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101402025-04-30T06:25:59.970ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101412025-04-30T06:25:59.970ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101422025-04-30T06:25:59.970ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
101432025-04-30T06:25:59.971ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101442025-04-30T06:25:59.971ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
101452025-04-30T06:25:59.971ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
101462025-04-30T06:25:59.972ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bb 87 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53600 req_id = 87b5e89b-1c6f-4d5e-b8e4-1ff7d43e97a6
101472025-04-30T06:25:59.972ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48007,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53600 req_id = 87b5e89b-1c6f-4d5e-b8e4-1ff7d43e97a6
101482025-04-30T06:25:59.972ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
101492025-04-30T06:25:59.973ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
101502025-04-30T06:25:59.974ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101512025-04-30T06:25:59.974ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53600 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 87b5e89b-1c6f-4d5e-b8e4-1ff7d43e97a6
101522025-04-30T06:25:59.974ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101532025-04-30T06:25:59.974ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
101542025-04-30T06:25:59.975ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101552025-04-30T06:25:59.975ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3224, tv_nsec: 774396901 } })
101562025-04-30T06:25:59.975ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
101572025-04-30T06:25:59.975ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
101582025-04-30T06:25:59.976ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 4c 0e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e L...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54461 req_id = dc4a1ed8-d464-4838-a5f5-3b530d5a9275
101592025-04-30T06:25:59.976ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19470,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54461 req_id = dc4a1ed8-d464-4838-a5f5-3b530d5a9275
101602025-04-30T06:25:59.976ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
101612025-04-30T06:25:59.976ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54461 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dc4a1ed8-d464-4838-a5f5-3b530d5a9275
101622025-04-30T06:25:59.976ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c6 ff 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60946 req_id = ee82a1c4-7357-4486-bdb3-0b004246a3e3
101632025-04-30T06:25:59.976ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50943,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60946 req_id = ee82a1c4-7357-4486-bdb3-0b004246a3e3
101642025-04-30T06:25:59.977ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
101652025-04-30T06:25:59.977ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60946 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = ee82a1c4-7357-4486-bdb3-0b004246a3e3
101662025-04-30T06:26:00.088ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
101672025-04-30T06:26:00.089ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2a 7e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c *~..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65400 req_id = 151f0b20-02a5-4f88-8737-7616902184a6
101682025-04-30T06:26:00.089ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10878,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65400 req_id = 151f0b20-02a5-4f88-8737-7616902184a6
101692025-04-30T06:26:00.089ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
101702025-04-30T06:26:00.089ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10878, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65400 req_id = 151f0b20-02a5-4f88-8737-7616902184a6
101712025-04-30T06:26:00.090ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
101722025-04-30T06:26:00.090ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
101732025-04-30T06:26:00.090ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
101742025-04-30T06:26:00.091ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7ba634f-1b1d-4bb3-9812-0890a16820a9", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101752025-04-30T06:26:00.092ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
101762025-04-30T06:26:00.093ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da407763-71a0-4949-9a4c-0b1080fd2968", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101772025-04-30T06:26:00.093ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
101782025-04-30T06:26:00.097ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cc36f21-725b-4ddf-b183-ff5b26325810", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101792025-04-30T06:26:00.097ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
101802025-04-30T06:26:00.098ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "73b0b7e9-ed91-402c-9d1d-4a3f123ea3a7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101812025-04-30T06:26:00.098ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
101822025-04-30T06:26:00.102ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f75eebbe-5b6d-4ae4-a12c-62e672502a97", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101832025-04-30T06:26:00.102ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
101842025-04-30T06:26:00.107ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f4401f-fd99-4d6c-9b5d-0727ca0089c7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101852025-04-30T06:26:00.107ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
101862025-04-30T06:26:00.109ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed923245-66eb-4480-bdc1-b962e40450e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101872025-04-30T06:26:00.110ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
101882025-04-30T06:26:00.112ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cefefe56-7be7-445e-a252-d41919428a43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101892025-04-30T06:26:00.112ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
101902025-04-30T06:26:00.116ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a43a9401-e40e-4af1-b85e-2a128d9af831", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101912025-04-30T06:26:00.117ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
101922025-04-30T06:26:00.118ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
101932025-04-30T06:26:00.119ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e124a25d-2540-476c-bac3-f12c12331afb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101942025-04-30T06:26:00.120ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
101952025-04-30T06:26:00.121ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d001c5b-6c7e-4f63-aa31-114d054a9740", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101962025-04-30T06:26:00.121ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
101972025-04-30T06:26:00.122ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cbfe3eb-e162-4a0a-924c-4eff5be9fb0d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
101982025-04-30T06:26:00.122ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
101992025-04-30T06:26:00.125ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4967248-73ed-43da-86fd-84ed04b16aa4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102002025-04-30T06:26:00.125ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
102012025-04-30T06:26:00.126ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6923bdbf-7157-40a7-bfa1-9edb5b085aa1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102022025-04-30T06:26:00.127ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
102032025-04-30T06:26:00.132ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b4791d8-e081-4793-a4e1-2553b9225701", "content-length": "111", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102042025-04-30T06:26:00.133ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
102052025-04-30T06:26:00.136ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03691a89-2161-4bf4-b554-3600d2c30023", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102062025-04-30T06:26:00.136ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
102072025-04-30T06:26:00.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56b54656-60b9-4b0c-af5b-1efe2cb3a16e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102082025-04-30T06:26:00.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
102092025-04-30T06:26:00.162ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfdf6527-ba70-4bf1-9886-8e4ab8ddffd6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102102025-04-30T06:26:00.162ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
102112025-04-30T06:26:00.171ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102122025-04-30T06:26:00.171ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102132025-04-30T06:26:00.171ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102142025-04-30T06:26:00.171ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
102152025-04-30T06:26:00.172ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102162025-04-30T06:26:00.176ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102172025-04-30T06:26:00.181ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102182025-04-30T06:26:00.181ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102192025-04-30T06:26:00.181ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
102202025-04-30T06:26:00.181ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102212025-04-30T06:26:00.183ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af712bf4-cc0d-44a5-ac61-4fb56091719d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102222025-04-30T06:26:00.183ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
102232025-04-30T06:26:00.184ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102242025-04-30T06:26:00.184ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
102252025-04-30T06:26:00.184ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102262025-04-30T06:26:00.185ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b24342-c0e1-4ed4-ba1e-c6ccca47abd3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102272025-04-30T06:26:00.185ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
102282025-04-30T06:26:00.187ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a57e060a-3d1a-4192-8d08-5a7adfe3a614", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102292025-04-30T06:26:00.187ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
102302025-04-30T06:26:00.188ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a9cc89-6f70-490d-9e2c-2ca8114c12e1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102312025-04-30T06:26:00.188ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
102322025-04-30T06:26:00.189ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d0ae079-2669-445a-b77f-5cdcbef9ab02", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102332025-04-30T06:26:00.190ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
102342025-04-30T06:26:00.191ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eee44ff-87b9-4387-bbc1-b80dbd722dae", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102352025-04-30T06:26:00.191ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
102362025-04-30T06:26:00.193ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cb297cf-c126-41a1-a5ca-e5b383bb4173", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102372025-04-30T06:26:00.193ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
102382025-04-30T06:26:00.195ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5804b7-0ccd-4d49-88d9-78ad25d0f898", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102392025-04-30T06:26:00.195ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
102402025-04-30T06:26:00.197ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef670543-6ab3-456a-8d02-13a1cf53a905", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102412025-04-30T06:26:00.197ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
102422025-04-30T06:26:00.200ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe5d87ed-3432-455b-b4b6-fdb60c3a95ed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102432025-04-30T06:26:00.200ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
102442025-04-30T06:26:00.203ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0be7e6bd-4a0f-4aae-b1b1-53f7761c5f1d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102452025-04-30T06:26:00.203ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
102462025-04-30T06:26:00.207ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21b4eb17-11c0-4ab4-9e95-b68bde4eaff4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102472025-04-30T06:26:00.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
102482025-04-30T06:26:00.211ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "95d3ece8-ab98-476a-969b-b4ebb057a9ef", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102492025-04-30T06:26:00.211ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
102502025-04-30T06:26:00.212ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
102512025-04-30T06:26:00.214ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15920113-9db1-4e1d-a2b1-1769518675c4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102522025-04-30T06:26:00.214ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
102532025-04-30T06:26:00.215ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75761e8a-f9a3-490e-871a-18f4149ac3d1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102542025-04-30T06:26:00.215ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
102552025-04-30T06:26:00.217ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94209951-cff3-4e5a-aaa4-165485dade72", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102562025-04-30T06:26:00.217ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
102572025-04-30T06:26:00.219ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f3d89f2-ac1a-4d97-b6a0-1940a93ece76", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102582025-04-30T06:26:00.220ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
102592025-04-30T06:26:00.222ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a795f88a-f7a5-4684-adc7-e541228236c4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102602025-04-30T06:26:00.222ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
102612025-04-30T06:26:00.227ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fa3cc41-2306-44f2-a147-07aef8a9edbf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102622025-04-30T06:26:00.227ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
102632025-04-30T06:26:00.228ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0281455-72e7-47b8-b675-42eaf450e7a1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102642025-04-30T06:26:00.228ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
102652025-04-30T06:26:00.233ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db4d02b8-c213-4c2b-bab4-41e1f914d66e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102662025-04-30T06:26:00.233ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
102672025-04-30T06:26:00.236ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "306458fd-c1d4-478f-b5d1-e6e8393d66b8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102682025-04-30T06:26:00.237ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
102692025-04-30T06:26:00.254ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f88e0922-73d3-48a4-89ec-57a9f8095486", "content-length": "178", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102702025-04-30T06:26:00.254ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
102712025-04-30T06:26:00.255ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5d91fae-ec1c-4237-b334-f9310297c59f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102722025-04-30T06:26:00.255ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
102732025-04-30T06:26:00.256ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f68ca85-7d9e-4e3b-b96f-a30dfcca520c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102742025-04-30T06:26:00.256ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
102752025-04-30T06:26:00.257ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5933adb6-5390-47e7-83ff-2405dad52629", "content-length": "729", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102762025-04-30T06:26:00.257ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
102772025-04-30T06:26:00.258ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e8555ec-cc35-48db-86ab-08c192ea38c7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102782025-04-30T06:26:00.258ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
102792025-04-30T06:26:00.259ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
102802025-04-30T06:26:00.261ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b0f4205-bb52-46f0-88de-3de394004522", "content-length": "835", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102812025-04-30T06:26:00.261ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
102822025-04-30T06:26:00.264ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102832025-04-30T06:26:00.264ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102842025-04-30T06:26:00.264ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102852025-04-30T06:26:00.264ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
102862025-04-30T06:26:00.273ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102872025-04-30T06:26:00.273ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9401181-9433-4a6c-866f-b036a087485e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102882025-04-30T06:26:00.273ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
102892025-04-30T06:26:00.276ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102902025-04-30T06:26:00.283ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102912025-04-30T06:26:00.283ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102922025-04-30T06:26:00.283ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
102932025-04-30T06:26:00.284ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102942025-04-30T06:26:00.286ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6ba9b2e-82d3-41b0-9a79-f93cb39a78c8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
102952025-04-30T06:26:00.286ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
102962025-04-30T06:26:00.287ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102972025-04-30T06:26:00.287ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
102982025-04-30T06:26:00.288ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102992025-04-30T06:26:00.290ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14d1a494-4d6e-4c5f-a602-a56f9a886739", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
103002025-04-30T06:26:00.290ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
103012025-04-30T06:26:00.292ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a507f4ad-f1f6-496c-b693-f54c37cf5bc8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
103022025-04-30T06:26:00.292ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103032025-04-30T06:26:00.292ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103042025-04-30T06:26:00.292ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103052025-04-30T06:26:00.292ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103062025-04-30T06:26:00.293ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103072025-04-30T06:26:00.295ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103082025-04-30T06:26:00.300ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103092025-04-30T06:26:00.300ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103102025-04-30T06:26:00.300ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103112025-04-30T06:26:00.300ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103122025-04-30T06:26:00.300ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103132025-04-30T06:26:00.301ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103142025-04-30T06:26:00.306ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103152025-04-30T06:26:00.315ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
103162025-04-30T06:26:00.315ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103172025-04-30T06:26:00.315ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103182025-04-30T06:26:00.316ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103192025-04-30T06:26:00.401ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
103202025-04-30T06:26:00.401ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
103212025-04-30T06:26:00.402ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:53857 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
103222025-04-30T06:26:00.403ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:53857 req_id = ee2002d6-2f35-426f-a22b-a114cd63afc7 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
103232025-04-30T06:26:00.403ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 747 local_addr = [::1]:42668 method = GET remote_addr = [::1]:53857 req_id = ee2002d6-2f35-426f-a22b-a114cd63afc7 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
103242025-04-30T06:26:00.414ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee2002d6-2f35-426f-a22b-a114cd63afc7", "content-length": "811", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
103252025-04-30T06:26:00.479ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
103262025-04-30T06:26:00.479ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
103272025-04-30T06:26:00.480ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:62327 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
103282025-04-30T06:26:00.480ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103292025-04-30T06:26:00.481ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103302025-04-30T06:26:00.482ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:62327 req_id = 043b509f-1b70-4679-8033-3e0e79bc1019 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
103312025-04-30T06:26:00.484ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1943 local_addr = [::1]:47918 method = GET remote_addr = [::1]:62327 req_id = 043b509f-1b70-4679-8033-3e0e79bc1019 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
103322025-04-30T06:26:00.486ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "043b509f-1b70-4679-8033-3e0e79bc1019", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
103332025-04-30T06:26:00.544ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
103342025-04-30T06:26:00.544ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
103352025-04-30T06:26:00.544ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103362025-04-30T06:26:00.545ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103372025-04-30T06:26:00.546ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103382025-04-30T06:26:00.546ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
103392025-04-30T06:26:00.547ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:36288 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
103402025-04-30T06:26:00.547ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:36288 req_id = ad1a0236-c204-475b-b34d-ddea91187251 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
103412025-04-30T06:26:00.552ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 4653 local_addr = [::1]:35744 method = GET remote_addr = [::1]:36288 req_id = ad1a0236-c204-475b-b34d-ddea91187251 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
103422025-04-30T06:26:00.553ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad1a0236-c204-475b-b34d-ddea91187251", "content-length": "439", "date": "Wed, 30 Apr 2025 06:25:59 GMT"} })
103432025-04-30T06:26:00.651ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
103442025-04-30T06:26:00.651ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
103452025-04-30T06:26:00.655ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:54170 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
103462025-04-30T06:26:00.657ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103472025-04-30T06:26:00.657ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103482025-04-30T06:26:00.666ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
103492025-04-30T06:26:00.668ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
103502025-04-30T06:26:00.672ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:54170 req_id = 6593e0bc-ff8b-46fd-a86b-aa86563e89bf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103512025-04-30T06:26:00.675ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3512 local_addr = [::1]:35343 method = GET remote_addr = [::1]:54170 req_id = 6593e0bc-ff8b-46fd-a86b-aa86563e89bf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
103522025-04-30T06:26:00.676ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
103532025-04-30T06:26:00.676ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
103542025-04-30T06:26:00.676ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
103552025-04-30T06:26:00.681ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6593e0bc-ff8b-46fd-a86b-aa86563e89bf", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
103562025-04-30T06:26:00.686ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103572025-04-30T06:26:00.686ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
103582025-04-30T06:26:00.686ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
103592025-04-30T06:26:00.693ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103602025-04-30T06:26:00.694ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103612025-04-30T06:26:00.694ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103622025-04-30T06:26:00.694ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103632025-04-30T06:26:00.695ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103642025-04-30T06:26:00.703ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103652025-04-30T06:26:00.712ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103662025-04-30T06:26:00.712ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103672025-04-30T06:26:00.712ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103682025-04-30T06:26:00.713ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103692025-04-30T06:26:00.714ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
103702025-04-30T06:26:00.724ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103712025-04-30T06:26:00.729ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103722025-04-30T06:26:00.729ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103732025-04-30T06:26:00.729ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103742025-04-30T06:26:00.729ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
103752025-04-30T06:26:00.731ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103762025-04-30T06:26:00.741ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103772025-04-30T06:26:00.747ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103782025-04-30T06:26:00.747ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103792025-04-30T06:26:00.747ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
103802025-04-30T06:26:00.748ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103812025-04-30T06:26:00.769ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
103822025-04-30T06:26:00.809ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
103832025-04-30T06:26:00.813ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
103842025-04-30T06:26:00.813ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 897.927121ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
103852025-04-30T06:26:00.814ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
103862025-04-30T06:26:00.814ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
103872025-04-30T06:26:00.814ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103882025-04-30T06:26:00.814ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103892025-04-30T06:26:00.814ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103902025-04-30T06:26:00.815ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103912025-04-30T06:26:00.817ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103922025-04-30T06:26:00.818ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
103932025-04-30T06:26:00.819ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103942025-04-30T06:26:00.820ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103952025-04-30T06:26:00.838ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
103962025-04-30T06:26:00.839ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103972025-04-30T06:26:00.839ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
103982025-04-30T06:26:00.839ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
103992025-04-30T06:26:00.846ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
104002025-04-30T06:26:00.847ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
104012025-04-30T06:26:00.849ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 48 25 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 H%..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51995 req_id = 95cb41e1-ad08-4b56-accb-3dc30c9cdb81
104022025-04-30T06:26:00.849ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18469,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51995 req_id = 95cb41e1-ad08-4b56-accb-3dc30c9cdb81
104032025-04-30T06:26:00.850ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
104042025-04-30T06:26:00.850ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
104052025-04-30T06:26:00.851ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104062025-04-30T06:26:00.851ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51995 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 95cb41e1-ad08-4b56-accb-3dc30c9cdb81
104072025-04-30T06:26:00.851ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3225, tv_nsec: 650721019 } })
104082025-04-30T06:26:00.851ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
104092025-04-30T06:26:00.852ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
104102025-04-30T06:26:00.852ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd be 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51515 req_id = 660c6711-9638-4ab3-b6db-d9f217b8a305
104112025-04-30T06:26:00.852ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48574,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51515 req_id = 660c6711-9638-4ab3-b6db-d9f217b8a305
104122025-04-30T06:26:00.853ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
104132025-04-30T06:26:00.853ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51515 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 660c6711-9638-4ab3-b6db-d9f217b8a305
104142025-04-30T06:26:00.853ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9e 85 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59547 req_id = 85feb663-d0c3-4d70-abb5-66f8bffd68ff
104152025-04-30T06:26:00.853ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40581,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59547 req_id = 85feb663-d0c3-4d70-abb5-66f8bffd68ff
104162025-04-30T06:26:00.853ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104172025-04-30T06:26:00.853ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59547 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 85feb663-d0c3-4d70-abb5-66f8bffd68ff
104182025-04-30T06:26:00.992ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
104192025-04-30T06:26:00.993ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 23 e1 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c #............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55093 req_id = 84b22a30-987b-4075-9751-208db2819b7c
104202025-04-30T06:26:00.994ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9185,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55093 req_id = 84b22a30-987b-4075-9751-208db2819b7c
104212025-04-30T06:26:00.995ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
104222025-04-30T06:26:00.995ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 9185, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55093 req_id = 84b22a30-987b-4075-9751-208db2819b7c
104232025-04-30T06:26:00.998ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
104242025-04-30T06:26:00.998ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
104252025-04-30T06:26:00.998ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
104262025-04-30T06:26:01.001ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4dab0ba6-0c39-40bb-b888-59f7ccc6e350", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104272025-04-30T06:26:01.001ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
104282025-04-30T06:26:01.005ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f57e0e39-b8db-4943-942e-6041c74b236e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104292025-04-30T06:26:01.005ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
104302025-04-30T06:26:01.008ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c82663b-85ed-496a-9019-f9f121f24e80", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104312025-04-30T06:26:01.008ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
104322025-04-30T06:26:01.011ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c584bb04-f153-410f-a529-8c7e63d55934", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104332025-04-30T06:26:01.011ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
104342025-04-30T06:26:01.016ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "562d4b04-77ef-4dde-a2dd-39c45ca5d5c2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104352025-04-30T06:26:01.017ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
104362025-04-30T06:26:01.017ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
104372025-04-30T06:26:01.020ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9961b89b-b130-459b-8503-067c5a3c15f0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104382025-04-30T06:26:01.020ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
104392025-04-30T06:26:01.021ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1379830-e3e9-407f-9dc3-18e4e54b960a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104402025-04-30T06:26:01.021ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
104412025-04-30T06:26:01.024ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bb04669-389c-4043-901b-b8ef78f9b3f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104422025-04-30T06:26:01.024ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
104432025-04-30T06:26:01.026ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56c2279e-47cd-4a8e-ad9d-c467d429ee96", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104442025-04-30T06:26:01.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
104452025-04-30T06:26:01.027ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6463658-f711-46a3-bcf0-583bf75fb249", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104462025-04-30T06:26:01.027ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
104472025-04-30T06:26:01.028ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dba7d0a-54ea-492f-baf0-6555f2b06af6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104482025-04-30T06:26:01.029ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
104492025-04-30T06:26:01.030ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e26b3180-77dd-4016-b05f-75a4f02e0fba", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104502025-04-30T06:26:01.030ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
104512025-04-30T06:26:01.032ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c6a4cf7-69d8-4c12-acc4-6cc83f6057f4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104522025-04-30T06:26:01.032ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
104532025-04-30T06:26:01.034ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3117fb31-199b-49ee-8be0-4788e8df9702", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104542025-04-30T06:26:01.035ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
104552025-04-30T06:26:01.040ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "417fc5ff-3e62-48de-ad1f-06fb5bf197e8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104562025-04-30T06:26:01.040ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
104572025-04-30T06:26:01.043ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3196a1cb-186e-4384-b1a9-2fbf56b3c1ef", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104582025-04-30T06:26:01.043ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
104592025-04-30T06:26:01.045ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5988b5ec-24da-4c5d-a72e-a6eed7b505c7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104602025-04-30T06:26:01.045ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
104612025-04-30T06:26:01.047ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b43696-8238-410c-a9c9-6c435d77b626", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104622025-04-30T06:26:01.047ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
104632025-04-30T06:26:01.050ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "326584fa-6a43-4159-9e03-f0de3b7f5846", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104642025-04-30T06:26:01.050ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
104652025-04-30T06:26:01.053ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df433eaf-9071-46e3-be3c-f3e052374aa6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104662025-04-30T06:26:01.053ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
104672025-04-30T06:26:01.055ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df63270f-aa11-4987-9eae-cbdda70dc610", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104682025-04-30T06:26:01.055ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
104692025-04-30T06:26:01.057ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36eed32-18dd-40b9-8169-150d8dd042c0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104702025-04-30T06:26:01.058ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
104712025-04-30T06:26:01.060ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50f8432e-a989-4ad6-b5d6-69884ccd5dfe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104722025-04-30T06:26:01.060ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
104732025-04-30T06:26:01.061ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4781a12-9fc8-4691-953e-bbcaa840df10", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104742025-04-30T06:26:01.061ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
104752025-04-30T06:26:01.064ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c03c8652-7f0c-4a61-af6d-60cf16a7be8e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104762025-04-30T06:26:01.064ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
104772025-04-30T06:26:01.067ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a378b8bb-9785-4c65-be69-df216c3618ae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104782025-04-30T06:26:01.067ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
104792025-04-30T06:26:01.068ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104802025-04-30T06:26:01.068ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104812025-04-30T06:26:01.068ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104822025-04-30T06:26:01.068ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
104832025-04-30T06:26:01.069ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104842025-04-30T06:26:01.071ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104852025-04-30T06:26:01.094ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104862025-04-30T06:26:01.094ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104872025-04-30T06:26:01.094ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
104882025-04-30T06:26:01.095ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104892025-04-30T06:26:01.095ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41e6ae57-0d30-489c-bff5-f928f84fb79e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104902025-04-30T06:26:01.096ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
104912025-04-30T06:26:01.099ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104922025-04-30T06:26:01.099ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
104932025-04-30T06:26:01.099ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104942025-04-30T06:26:01.101ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6dd84c1-0c78-485c-b887-c979b4ca1f20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104952025-04-30T06:26:01.101ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
104962025-04-30T06:26:01.105ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7dff906-835e-4993-8edd-0789e8026c54", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104972025-04-30T06:26:01.106ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
104982025-04-30T06:26:01.108ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "226fc704-0a0f-461f-a124-d98fadba2b60", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
104992025-04-30T06:26:01.109ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
105002025-04-30T06:26:01.111ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aea07579-cf46-4cc4-96f1-ce7d14b67c4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105012025-04-30T06:26:01.112ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
105022025-04-30T06:26:01.115ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b82d4850-35f4-4ae7-9a50-7f49328a9be0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105032025-04-30T06:26:01.115ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
105042025-04-30T06:26:01.117ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "148d4730-0ccf-4549-8999-e80a2a5db1ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105052025-04-30T06:26:01.117ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
105062025-04-30T06:26:01.118ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6799a96d-e1c5-4639-ae11-4c15e55abc10", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105072025-04-30T06:26:01.118ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
105082025-04-30T06:26:01.122ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3df498c8-4cd2-4f47-94cc-929f67a26eb0", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105092025-04-30T06:26:01.123ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
105102025-04-30T06:26:01.126ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20942365-92dc-4627-8772-6767d86b3888", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105112025-04-30T06:26:01.126ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
105122025-04-30T06:26:01.131ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
105132025-04-30T06:26:01.133ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bd65e71-6fee-41a2-b557-458160e5b619", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105142025-04-30T06:26:01.135ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
105152025-04-30T06:26:01.140ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ed0fdbc-de0a-4074-bc0c-09200c963d0f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105162025-04-30T06:26:01.141ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
105172025-04-30T06:26:01.173ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8219793-160f-4dff-b49c-01edcdad0b2e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105182025-04-30T06:26:01.173ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
105192025-04-30T06:26:01.184ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105202025-04-30T06:26:01.184ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105212025-04-30T06:26:01.184ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105222025-04-30T06:26:01.184ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
105232025-04-30T06:26:01.185ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105242025-04-30T06:26:01.188ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105252025-04-30T06:26:01.196ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105262025-04-30T06:26:01.196ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105272025-04-30T06:26:01.196ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
105282025-04-30T06:26:01.196ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105292025-04-30T06:26:01.201ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ab63b6b-e3d1-42ad-99f0-3fda20146f9b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105302025-04-30T06:26:01.202ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
105312025-04-30T06:26:01.202ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105322025-04-30T06:26:01.202ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
105332025-04-30T06:26:01.203ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105342025-04-30T06:26:01.206ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd63cd09-80b3-4379-8963-bc6b7ae31bd6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105352025-04-30T06:26:01.206ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
105362025-04-30T06:26:01.208ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58d5a8e6-6c5d-45da-a402-0e6cf12d0a0f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105372025-04-30T06:26:01.208ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
105382025-04-30T06:26:01.210ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2afc503c-2464-47ce-adac-120b36812856", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105392025-04-30T06:26:01.210ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
105402025-04-30T06:26:01.211ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0547dff9-9776-4a51-b6f8-79171ff5a28e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105412025-04-30T06:26:01.211ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
105422025-04-30T06:26:01.214ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecc37eb9-1e88-4a80-8072-0fc8f06729e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105432025-04-30T06:26:01.214ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
105442025-04-30T06:26:01.214ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
105452025-04-30T06:26:01.215ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "443df485-2054-467f-b31f-e521c6eb0e68", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105462025-04-30T06:26:01.216ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
105472025-04-30T06:26:01.217ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "94a67f70-6c3e-4e4b-86b8-9e90e29a2240", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105482025-04-30T06:26:01.217ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
105492025-04-30T06:26:01.219ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "036eb554-63d1-44ac-aa12-dbdeaf259d4d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105502025-04-30T06:26:01.225ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
105512025-04-30T06:26:01.231ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1358db19-8a5d-42a2-8e94-611755f76eaf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105522025-04-30T06:26:01.231ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
105532025-04-30T06:26:01.247ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1954cc9d-cd57-4b11-a054-ecdfcde13318", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105542025-04-30T06:26:01.248ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105552025-04-30T06:26:01.248ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105562025-04-30T06:26:01.248ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105572025-04-30T06:26:01.248ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105582025-04-30T06:26:01.254ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105592025-04-30T06:26:01.257ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105602025-04-30T06:26:01.260ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105612025-04-30T06:26:01.260ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
105622025-04-30T06:26:01.260ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105632025-04-30T06:26:01.260ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105642025-04-30T06:26:01.260ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105652025-04-30T06:26:01.261ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105662025-04-30T06:26:01.267ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105672025-04-30T06:26:01.274ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
105682025-04-30T06:26:01.274ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
105692025-04-30T06:26:01.274ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
105702025-04-30T06:26:01.274ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
105712025-04-30T06:26:01.360ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
105722025-04-30T06:26:01.360ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
105732025-04-30T06:26:01.362ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:55788 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
105742025-04-30T06:26:01.362ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
105752025-04-30T06:26:01.362ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105762025-04-30T06:26:01.370ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
105772025-04-30T06:26:01.371ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:55788 req_id = d24e6413-83be-4367-8147-c66154177319 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
105782025-04-30T06:26:01.371ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 47662742-6d08-4458-ba6d-d1c57c22e1ea uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105792025-04-30T06:26:01.373ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 47662742-6d08-4458-ba6d-d1c57c22e1ea resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105802025-04-30T06:26:01.373ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 47662742-6d08-4458-ba6d-d1c57c22e1ea roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105812025-04-30T06:26:01.374ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 47662742-6d08-4458-ba6d-d1c57c22e1ea resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105822025-04-30T06:26:01.374ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3671 local_addr = [::1]:42668 method = GET remote_addr = [::1]:55788 req_id = d24e6413-83be-4367-8147-c66154177319 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
105832025-04-30T06:26:01.376ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d24e6413-83be-4367-8147-c66154177319", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105842025-04-30T06:26:01.504ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
105852025-04-30T06:26:01.508ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
105862025-04-30T06:26:01.508ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105872025-04-30T06:26:01.508ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105882025-04-30T06:26:01.508ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105892025-04-30T06:26:01.508ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
105902025-04-30T06:26:01.510ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105912025-04-30T06:26:01.510ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:54526 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
105922025-04-30T06:26:01.510ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105932025-04-30T06:26:01.510ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105942025-04-30T06:26:01.511ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 140475 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 47662742-6d08-4458-ba6d-d1c57c22e1ea response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105952025-04-30T06:26:01.512ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:54526 req_id = 7cd80267-e6d7-4ae2-bf5b-44ed4e4e9374 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
105962025-04-30T06:26:01.518ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "47662742-6d08-4458-ba6d-d1c57c22e1ea", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
105972025-04-30T06:26:01.518ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
105982025-04-30T06:26:01.520ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 7701 local_addr = [::1]:47918 method = GET remote_addr = [::1]:54526 req_id = 7cd80267-e6d7-4ae2-bf5b-44ed4e4e9374 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
105992025-04-30T06:26:01.522ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 8a031330-7ffc-4bdf-af71-1f01dcc4adc0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
106002025-04-30T06:26:01.523ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 8a031330-7ffc-4bdf-af71-1f01dcc4adc0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
106012025-04-30T06:26:01.523ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 8a031330-7ffc-4bdf-af71-1f01dcc4adc0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
106022025-04-30T06:26:01.524ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 8a031330-7ffc-4bdf-af71-1f01dcc4adc0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
106032025-04-30T06:26:01.524ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cd80267-e6d7-4ae2-bf5b-44ed4e4e9374", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
106042025-04-30T06:26:01.575ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
106052025-04-30T06:26:01.575ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
106062025-04-30T06:26:01.576ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106072025-04-30T06:26:01.578ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106082025-04-30T06:26:01.578ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106092025-04-30T06:26:01.578ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106102025-04-30T06:26:01.579ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106112025-04-30T06:26:01.585ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106122025-04-30T06:26:01.585ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106132025-04-30T06:26:01.585ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
106142025-04-30T06:26:01.586ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106152025-04-30T06:26:01.587ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:43487 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
106162025-04-30T06:26:01.592ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:43487 req_id = c5ae6276-387c-4444-9b18-37acf76820c6 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
106172025-04-30T06:26:01.609ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 17915 local_addr = [::1]:35744 method = GET remote_addr = [::1]:43487 req_id = c5ae6276-387c-4444-9b18-37acf76820c6 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
106182025-04-30T06:26:01.610ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 87553 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 8a031330-7ffc-4bdf-af71-1f01dcc4adc0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
106192025-04-30T06:26:01.612ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106202025-04-30T06:26:01.612ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
106212025-04-30T06:26:01.613ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106222025-04-30T06:26:01.613ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106232025-04-30T06:26:01.613ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106242025-04-30T06:26:01.614ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5ae6276-387c-4444-9b18-37acf76820c6", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
106252025-04-30T06:26:01.671ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
106262025-04-30T06:26:01.671ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
106272025-04-30T06:26:01.671ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a031330-7ffc-4bdf-af71-1f01dcc4adc0", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:00 GMT"} })
106282025-04-30T06:26:01.671ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106292025-04-30T06:26:01.671ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
106302025-04-30T06:26:01.671ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
106312025-04-30T06:26:01.672ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
106322025-04-30T06:26:01.673ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
106332025-04-30T06:26:01.674ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:49412 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
106342025-04-30T06:26:01.682ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:49412 req_id = d7918753-a851-4916-aa7f-bdb8b96392c9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106352025-04-30T06:26:01.683ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 726 local_addr = [::1]:35343 method = GET remote_addr = [::1]:49412 req_id = d7918753-a851-4916-aa7f-bdb8b96392c9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106362025-04-30T06:26:01.690ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7918753-a851-4916-aa7f-bdb8b96392c9", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
106372025-04-30T06:26:01.692ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106382025-04-30T06:26:01.692ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106392025-04-30T06:26:01.692ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
106402025-04-30T06:26:01.697ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106412025-04-30T06:26:01.697ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106422025-04-30T06:26:01.697ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106432025-04-30T06:26:01.697ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106442025-04-30T06:26:01.698ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106452025-04-30T06:26:01.700ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
106462025-04-30T06:26:01.700ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
106472025-04-30T06:26:01.703ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106482025-04-30T06:26:01.710ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106492025-04-30T06:26:01.711ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106502025-04-30T06:26:01.711ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106512025-04-30T06:26:01.711ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106522025-04-30T06:26:01.720ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
106532025-04-30T06:26:01.720ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106542025-04-30T06:26:01.753ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
106552025-04-30T06:26:01.755ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
106562025-04-30T06:26:01.756ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
106572025-04-30T06:26:01.756ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 942.051447ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
106582025-04-30T06:26:01.756ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
106592025-04-30T06:26:01.756ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106602025-04-30T06:26:01.756ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106612025-04-30T06:26:01.756ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106622025-04-30T06:26:01.756ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106632025-04-30T06:26:01.757ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106642025-04-30T06:26:01.760ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106652025-04-30T06:26:01.765ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106662025-04-30T06:26:01.765ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106672025-04-30T06:26:01.765ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106682025-04-30T06:26:01.766ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106692025-04-30T06:26:01.770ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
106702025-04-30T06:26:01.770ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
106712025-04-30T06:26:01.771ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cf 3c 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .<..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50474 req_id = 4abf05a8-3d2d-4934-bf5d-43302ebe1a8e
106722025-04-30T06:26:01.771ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53052,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50474 req_id = 4abf05a8-3d2d-4934-bf5d-43302ebe1a8e
106732025-04-30T06:26:01.771ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
106742025-04-30T06:26:01.772ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
106752025-04-30T06:26:01.772ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106762025-04-30T06:26:01.772ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50474 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4abf05a8-3d2d-4934-bf5d-43302ebe1a8e
106772025-04-30T06:26:01.772ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3226, tv_nsec: 571776653 } })
106782025-04-30T06:26:01.772ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
106792025-04-30T06:26:01.773ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
106802025-04-30T06:26:01.773ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
106812025-04-30T06:26:01.773ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c3 8d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54819 req_id = 194d8b4e-78eb-4864-8621-dc1ed6a02446
106822025-04-30T06:26:01.773ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50061,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54819 req_id = 194d8b4e-78eb-4864-8621-dc1ed6a02446
106832025-04-30T06:26:01.773ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
106842025-04-30T06:26:01.773ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54819 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 194d8b4e-78eb-4864-8621-dc1ed6a02446
106852025-04-30T06:26:01.774ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 64 61 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e da..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55766 req_id = 4f5dc416-dbab-4fe9-b50f-fc40ac8aad82
106862025-04-30T06:26:01.774ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25697,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55766 req_id = 4f5dc416-dbab-4fe9-b50f-fc40ac8aad82
106872025-04-30T06:26:01.774ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
106882025-04-30T06:26:01.774ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55766 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4f5dc416-dbab-4fe9-b50f-fc40ac8aad82
106892025-04-30T06:26:01.903ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
106902025-04-30T06:26:01.904ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106912025-04-30T06:26:01.904ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106922025-04-30T06:26:01.904ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106932025-04-30T06:26:01.904ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
106942025-04-30T06:26:01.906ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106952025-04-30T06:26:01.907ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6c d3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c l............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59527 req_id = 4940453d-e43c-4b38-8260-b08b1280459c
106962025-04-30T06:26:01.907ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27859,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59527 req_id = 4940453d-e43c-4b38-8260-b08b1280459c
106972025-04-30T06:26:01.907ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
106982025-04-30T06:26:01.907ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 27859, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59527 req_id = 4940453d-e43c-4b38-8260-b08b1280459c
106992025-04-30T06:26:01.908ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
107002025-04-30T06:26:01.908ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
107012025-04-30T06:26:01.908ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
107022025-04-30T06:26:01.910ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107032025-04-30T06:26:01.919ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107042025-04-30T06:26:01.919ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107052025-04-30T06:26:01.919ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
107062025-04-30T06:26:01.920ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107072025-04-30T06:26:01.925ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "37006721-87e7-466d-9608-7df67c1fded6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107082025-04-30T06:26:01.926ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
107092025-04-30T06:26:01.926ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107102025-04-30T06:26:01.926ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
107112025-04-30T06:26:01.927ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107122025-04-30T06:26:01.933ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "731095af-9b51-4d5f-bc89-d594b83e749e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107132025-04-30T06:26:01.933ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
107142025-04-30T06:26:01.936ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f2066bc-061b-40f3-bb27-138aa1888692", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107152025-04-30T06:26:01.936ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
107162025-04-30T06:26:01.944ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "000d896e-d5ba-4269-8638-dff7911d3942", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107172025-04-30T06:26:01.944ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
107182025-04-30T06:26:01.948ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f62cfd5c-af05-4b23-a24b-6c58bac29c3f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107192025-04-30T06:26:01.948ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
107202025-04-30T06:26:01.953ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95ed171d-377b-49df-982a-8d31675c0de6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107212025-04-30T06:26:01.953ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
107222025-04-30T06:26:01.965ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c808f869-506e-463a-997b-1f26b68e0eee", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107232025-04-30T06:26:01.965ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
107242025-04-30T06:26:01.971ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f92547a7-9f71-430d-821f-68fe86e9628f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107252025-04-30T06:26:01.971ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
107262025-04-30T06:26:01.972ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
107272025-04-30T06:26:01.972ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26d907d7-186a-400d-af6f-203725592584", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107282025-04-30T06:26:01.972ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
107292025-04-30T06:26:01.974ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa653e27-89ed-433b-88a4-0750c730eb43", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107302025-04-30T06:26:01.975ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
107312025-04-30T06:26:01.976ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64377f6a-7469-4355-96a5-bef18d9b662e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107322025-04-30T06:26:01.976ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
107332025-04-30T06:26:01.977ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e12734b0-e500-441d-8d02-d8c645a48fc2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107342025-04-30T06:26:01.977ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
107352025-04-30T06:26:01.980ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20b46316-f4a7-4f3b-b542-3bb9cb67b330", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107362025-04-30T06:26:01.980ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
107372025-04-30T06:26:01.984ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c52c59a7-d465-480d-9d70-b8881013ee11", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107382025-04-30T06:26:01.984ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
107392025-04-30T06:26:01.989ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6350fbe-015d-4c48-9a7e-3fecdaf0e084", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107402025-04-30T06:26:01.989ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
107412025-04-30T06:26:01.992ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e72871e-e9a5-442b-9b36-fd3e4100838e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107422025-04-30T06:26:01.992ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
107432025-04-30T06:26:01.996ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7915996b-5b03-4938-abb2-e1ef55d8f469", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107442025-04-30T06:26:01.996ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
107452025-04-30T06:26:01.999ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4470314-ea9e-4ad7-a8f9-772773e9e971", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107462025-04-30T06:26:01.999ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
107472025-04-30T06:26:02.003ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e01f53e5-8ea8-4269-8ca0-6673c649ff7b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107482025-04-30T06:26:02.003ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
107492025-04-30T06:26:02.004ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bd42dba-a2df-4fce-a72f-0ab76540eb42", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107502025-04-30T06:26:02.005ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
107512025-04-30T06:26:02.006ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2277c912-a590-4a67-bd3c-e0bd3125647b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107522025-04-30T06:26:02.006ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
107532025-04-30T06:26:02.009ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0026024b-8da9-4516-a81e-cd833be7a480", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107542025-04-30T06:26:02.010ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
107552025-04-30T06:26:02.011ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd09a5a9-7860-44af-a6f6-2d62c8d08c10", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107562025-04-30T06:26:02.012ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
107572025-04-30T06:26:02.013ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d366aee6-5802-4fd1-be7e-5f0c2615c95b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107582025-04-30T06:26:02.014ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
107592025-04-30T06:26:02.017ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed94edf1-658d-4f20-a694-79fc91974ff2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107602025-04-30T06:26:02.018ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
107612025-04-30T06:26:02.021ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f056de06-76c8-490d-93b3-b5a7a2b83ce0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107622025-04-30T06:26:02.021ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
107632025-04-30T06:26:02.023ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107642025-04-30T06:26:02.023ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107652025-04-30T06:26:02.023ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107662025-04-30T06:26:02.023ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
107672025-04-30T06:26:02.024ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107682025-04-30T06:26:02.026ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb450413-b574-401d-8e12-a5fc9779fa2c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107692025-04-30T06:26:02.026ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
107702025-04-30T06:26:02.031ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107712025-04-30T06:26:02.038ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107722025-04-30T06:26:02.039ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107732025-04-30T06:26:02.039ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
107742025-04-30T06:26:02.039ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107752025-04-30T06:26:02.041ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1676f0aa-ed5d-4295-ac2c-14c3ae5436ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107762025-04-30T06:26:02.042ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
107772025-04-30T06:26:02.042ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107782025-04-30T06:26:02.042ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
107792025-04-30T06:26:02.044ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107802025-04-30T06:26:02.051ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37728d53-c285-45a9-ba06-28c17f7bfca5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107812025-04-30T06:26:02.052ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
107822025-04-30T06:26:02.058ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd7e4a29-d597-4192-a456-7cca8c6cca58", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107832025-04-30T06:26:02.059ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
107842025-04-30T06:26:02.063ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f105ce40-66d8-48e5-9b3c-f508bed3c061", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107852025-04-30T06:26:02.064ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
107862025-04-30T06:26:02.065ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6268d204-a6ff-4932-87f7-defacf709eb8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107872025-04-30T06:26:02.068ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
107882025-04-30T06:26:02.074ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e6cc23-024b-4d43-81b8-7118dc0b78f7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107892025-04-30T06:26:02.074ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
107902025-04-30T06:26:02.078ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbddd461-43d6-4d42-9211-fb0fd81a82a2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107912025-04-30T06:26:02.078ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
107922025-04-30T06:26:02.080ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a7dd1bf-159f-4dd9-87b6-4cc7758f9a4c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107932025-04-30T06:26:02.081ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
107942025-04-30T06:26:02.088ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
107952025-04-30T06:26:02.091ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0545fb7-1a05-43cf-b95e-3cdac7b8cb9d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107962025-04-30T06:26:02.101ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
107972025-04-30T06:26:02.105ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9414d293-96d7-42b4-81ac-9bfa749b4ec6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
107982025-04-30T06:26:02.105ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
107992025-04-30T06:26:02.108ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71708aee-1c4c-4652-8f41-b4f89499540f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108002025-04-30T06:26:02.108ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
108012025-04-30T06:26:02.111ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "540192fe-400d-4709-86f0-6d213ea056c3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108022025-04-30T06:26:02.111ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
108032025-04-30T06:26:02.113ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80cacf29-1346-443e-9c33-0b6b29bce132", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108042025-04-30T06:26:02.115ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
108052025-04-30T06:26:02.117ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d90bb64-4245-492d-8be7-79408f88f6eb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108062025-04-30T06:26:02.117ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
108072025-04-30T06:26:02.119ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b5ef97c-fc01-4a85-a95c-b6b9fc339dbe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108082025-04-30T06:26:02.119ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
108092025-04-30T06:26:02.120ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "705910cf-8d2b-4470-9668-cb14675fb286", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108102025-04-30T06:26:02.120ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
108112025-04-30T06:26:02.122ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98fabf53-463e-40a7-a3db-ded515071f63", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108122025-04-30T06:26:02.122ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
108132025-04-30T06:26:02.125ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed5e245-5236-49e4-b82a-f470323bd075", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108142025-04-30T06:26:02.125ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
108152025-04-30T06:26:02.125ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
108162025-04-30T06:26:02.127ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bbde876-9429-4e4b-830f-2bc9cdba7f92", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108172025-04-30T06:26:02.127ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
108182025-04-30T06:26:02.128ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "187274cc-b59f-4c6a-93da-1d0c3e3f493e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108192025-04-30T06:26:02.129ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
108202025-04-30T06:26:02.130ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20131685-517e-4b07-83d8-62d916933b95", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108212025-04-30T06:26:02.130ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
108222025-04-30T06:26:02.133ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "400fa137-d2d3-42c9-b90f-b6d62fa26d49", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108232025-04-30T06:26:02.134ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
108242025-04-30T06:26:02.135ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b02a762a-d3ac-4938-a9bb-4a437886e8ad", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108252025-04-30T06:26:02.135ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108262025-04-30T06:26:02.135ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108272025-04-30T06:26:02.135ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108282025-04-30T06:26:02.135ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108292025-04-30T06:26:02.137ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108302025-04-30T06:26:02.139ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108312025-04-30T06:26:02.139ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108322025-04-30T06:26:02.139ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108332025-04-30T06:26:02.139ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
108342025-04-30T06:26:02.140ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108352025-04-30T06:26:02.141ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108362025-04-30T06:26:02.147ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108372025-04-30T06:26:02.147ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108382025-04-30T06:26:02.147ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108392025-04-30T06:26:02.147ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108402025-04-30T06:26:02.147ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108412025-04-30T06:26:02.147ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108422025-04-30T06:26:02.151ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108432025-04-30T06:26:02.157ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108442025-04-30T06:26:02.158ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108452025-04-30T06:26:02.158ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
108462025-04-30T06:26:02.159ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108472025-04-30T06:26:02.161ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108482025-04-30T06:26:02.164ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108492025-04-30T06:26:02.164ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108502025-04-30T06:26:02.164ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108512025-04-30T06:26:02.164ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108522025-04-30T06:26:02.168ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108532025-04-30T06:26:02.168ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
108542025-04-30T06:26:02.169ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108552025-04-30T06:26:02.232ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
108562025-04-30T06:26:02.232ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
108572025-04-30T06:26:02.234ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:37825 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
108582025-04-30T06:26:02.236ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:37825 req_id = 5d751e06-73cf-4b17-80cc-6eaf64d617b8 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
108592025-04-30T06:26:02.244ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 8675 local_addr = [::1]:42668 method = GET remote_addr = [::1]:37825 req_id = 5d751e06-73cf-4b17-80cc-6eaf64d617b8 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
108602025-04-30T06:26:02.245ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d751e06-73cf-4b17-80cc-6eaf64d617b8", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108612025-04-30T06:26:02.321ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
108622025-04-30T06:26:02.322ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
108632025-04-30T06:26:02.328ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:62754 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
108642025-04-30T06:26:02.340ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:62754 req_id = 80463781-b143-4c45-8675-f4e4512b5578 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
108652025-04-30T06:26:02.344ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1275 local_addr = [::1]:47918 method = GET remote_addr = [::1]:62754 req_id = 80463781-b143-4c45-8675-f4e4512b5578 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
108662025-04-30T06:26:02.348ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "80463781-b143-4c45-8675-f4e4512b5578", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108672025-04-30T06:26:02.438ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
108682025-04-30T06:26:02.438ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
108692025-04-30T06:26:02.441ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:59530 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
108702025-04-30T06:26:02.441ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108712025-04-30T06:26:02.441ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108722025-04-30T06:26:02.444ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:59530 req_id = 232acdc8-4794-43d0-8e5f-dc050ecdbc39 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
108732025-04-30T06:26:02.449ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 5121 local_addr = [::1]:35744 method = GET remote_addr = [::1]:59530 req_id = 232acdc8-4794-43d0-8e5f-dc050ecdbc39 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
108742025-04-30T06:26:02.452ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "232acdc8-4794-43d0-8e5f-dc050ecdbc39", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108752025-04-30T06:26:02.523ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
108762025-04-30T06:26:02.523ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
108772025-04-30T06:26:02.523ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108782025-04-30T06:26:02.524ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108792025-04-30T06:26:02.524ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108802025-04-30T06:26:02.524ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108812025-04-30T06:26:02.534ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:37203 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
108822025-04-30T06:26:02.538ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:37203 req_id = a2b7992d-065e-4724-81ac-9589aac6465f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108832025-04-30T06:26:02.548ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 9864 local_addr = [::1]:35343 method = GET remote_addr = [::1]:37203 req_id = a2b7992d-065e-4724-81ac-9589aac6465f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
108842025-04-30T06:26:02.554ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2b7992d-065e-4724-81ac-9589aac6465f", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:01 GMT"} })
108852025-04-30T06:26:02.559ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108862025-04-30T06:26:02.559ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
108872025-04-30T06:26:02.560ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
108882025-04-30T06:26:02.566ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
108892025-04-30T06:26:02.566ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108902025-04-30T06:26:02.566ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108912025-04-30T06:26:02.566ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108922025-04-30T06:26:02.567ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108932025-04-30T06:26:02.581ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108942025-04-30T06:26:02.585ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
108952025-04-30T06:26:02.585ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108962025-04-30T06:26:02.585ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108972025-04-30T06:26:02.586ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108982025-04-30T06:26:02.602ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
108992025-04-30T06:26:02.611ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109002025-04-30T06:26:02.611ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109012025-04-30T06:26:02.611ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
109022025-04-30T06:26:02.612ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
109032025-04-30T06:26:02.618ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
109042025-04-30T06:26:02.618ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
109052025-04-30T06:26:02.623ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
109062025-04-30T06:26:02.623ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
109072025-04-30T06:26:02.631ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
109082025-04-30T06:26:02.631ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
109092025-04-30T06:26:02.631ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 875.082655ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
109102025-04-30T06:26:02.631ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
109112025-04-30T06:26:02.631ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109122025-04-30T06:26:02.631ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109132025-04-30T06:26:02.631ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109142025-04-30T06:26:02.631ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109152025-04-30T06:26:02.632ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109162025-04-30T06:26:02.638ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109172025-04-30T06:26:02.642ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109182025-04-30T06:26:02.643ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109192025-04-30T06:26:02.643ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109202025-04-30T06:26:02.643ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109212025-04-30T06:26:02.646ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
109222025-04-30T06:26:02.646ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
109232025-04-30T06:26:02.647ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c7 54 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .T..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63939 req_id = bc16e975-0c35-477e-9212-2c3659f81fc2
109242025-04-30T06:26:02.647ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51028,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63939 req_id = bc16e975-0c35-477e-9212-2c3659f81fc2
109252025-04-30T06:26:02.647ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
109262025-04-30T06:26:02.647ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
109272025-04-30T06:26:02.648ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109282025-04-30T06:26:02.648ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63939 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = bc16e975-0c35-477e-9212-2c3659f81fc2
109292025-04-30T06:26:02.649ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3227, tv_nsec: 448239328 } })
109302025-04-30T06:26:02.649ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
109312025-04-30T06:26:02.649ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
109322025-04-30T06:26:02.649ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9d 42 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .B..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50617 req_id = 730f6ce6-156e-47cb-b3be-10d2a7ebd9d1
109332025-04-30T06:26:02.649ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40258,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50617 req_id = 730f6ce6-156e-47cb-b3be-10d2a7ebd9d1
109342025-04-30T06:26:02.650ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
109352025-04-30T06:26:02.650ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50617 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 730f6ce6-156e-47cb-b3be-10d2a7ebd9d1
109362025-04-30T06:26:02.650ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0e a5 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53137 req_id = 739a19e9-002e-4187-8e8c-8369f11e7d79
109372025-04-30T06:26:02.650ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3749,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53137 req_id = 739a19e9-002e-4187-8e8c-8369f11e7d79
109382025-04-30T06:26:02.650ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109392025-04-30T06:26:02.650ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53137 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 739a19e9-002e-4187-8e8c-8369f11e7d79
109402025-04-30T06:26:02.766ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
109412025-04-30T06:26:02.767ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109422025-04-30T06:26:02.767ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109432025-04-30T06:26:02.767ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109442025-04-30T06:26:02.767ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
109452025-04-30T06:26:02.770ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109462025-04-30T06:26:02.770ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
109472025-04-30T06:26:02.770ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: cb ce 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58466 req_id = 0e237d57-fdcf-4d81-b372-a96caaa3b6d1
109482025-04-30T06:26:02.770ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52174,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58466 req_id = 0e237d57-fdcf-4d81-b372-a96caaa3b6d1
109492025-04-30T06:26:02.772ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
109502025-04-30T06:26:02.772ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 52174, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58466 req_id = 0e237d57-fdcf-4d81-b372-a96caaa3b6d1
109512025-04-30T06:26:02.773ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
109522025-04-30T06:26:02.773ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
109532025-04-30T06:26:02.773ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
109542025-04-30T06:26:02.774ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109552025-04-30T06:26:02.791ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109562025-04-30T06:26:02.791ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109572025-04-30T06:26:02.791ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
109582025-04-30T06:26:02.792ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109592025-04-30T06:26:02.795ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a5405a0-719d-4252-8c69-71691a920993", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109602025-04-30T06:26:02.795ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
109612025-04-30T06:26:02.796ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109622025-04-30T06:26:02.796ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
109632025-04-30T06:26:02.797ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109642025-04-30T06:26:02.800ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca3d7a09-3d66-47f2-88b4-d951bf57cbe5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109652025-04-30T06:26:02.800ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
109662025-04-30T06:26:02.813ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4ca9f4c-8b96-4878-bff7-a367d9e650e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109672025-04-30T06:26:02.813ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
109682025-04-30T06:26:02.820ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b144d8a-089a-410b-9cbd-bd44c832fc6b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109692025-04-30T06:26:02.820ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
109702025-04-30T06:26:02.829ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbbb4794-f52c-47ad-9246-7af615a5ce2b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109712025-04-30T06:26:02.830ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
109722025-04-30T06:26:02.835ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d46c4a83-d374-4d6c-a02b-012d44e53229", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109732025-04-30T06:26:02.835ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
109742025-04-30T06:26:02.837ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f785fc-6e1b-42fb-ad9c-a739e798e378", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109752025-04-30T06:26:02.837ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
109762025-04-30T06:26:02.840ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cc3a1e1-825d-48b4-a4e9-05aecd842325", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109772025-04-30T06:26:02.840ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
109782025-04-30T06:26:02.842ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ba750e2-203f-48d0-b0cb-d0d2b479e9ba", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109792025-04-30T06:26:02.842ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
109802025-04-30T06:26:02.843ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "65b5e025-194d-4abc-ab11-182b261e0aca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109812025-04-30T06:26:02.844ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
109822025-04-30T06:26:02.847ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "47d961ee-0f37-401d-9615-aba5822c72f5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109832025-04-30T06:26:02.847ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
109842025-04-30T06:26:02.848ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
109852025-04-30T06:26:02.848ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf096617-460d-45d0-947b-32e0c545bf36", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109862025-04-30T06:26:02.848ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
109872025-04-30T06:26:02.850ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65175b1a-c209-488f-8a6b-6a2834147250", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109882025-04-30T06:26:02.850ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
109892025-04-30T06:26:02.852ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "948685dd-63c7-45bb-a0a2-27ec8f48290b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109902025-04-30T06:26:02.852ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
109912025-04-30T06:26:02.854ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f03856d7-ef10-4d7d-9e47-00c4dc878ea5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109922025-04-30T06:26:02.854ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
109932025-04-30T06:26:02.856ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e70851d-9d6c-4993-9a2b-cb5ed92a6736", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109942025-04-30T06:26:02.856ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
109952025-04-30T06:26:02.858ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39beb59e-6b63-4d5f-ab26-09eb5354390f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109962025-04-30T06:26:02.858ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
109972025-04-30T06:26:02.859ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0319c9f-5cd6-42f2-a909-4db5e95c7225", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
109982025-04-30T06:26:02.860ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
109992025-04-30T06:26:02.862ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "125bd721-67ad-4263-9c9e-ddace0972c2b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110002025-04-30T06:26:02.862ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
110012025-04-30T06:26:02.863ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21c20478-760e-4a9c-b2c5-4557a506b17e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110022025-04-30T06:26:02.863ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
110032025-04-30T06:26:02.864ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae2c5024-fdf6-40fa-9c7b-9586b55ff470", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110042025-04-30T06:26:02.864ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
110052025-04-30T06:26:02.865ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cde30815-90c5-4106-a162-a11cf234c9f1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110062025-04-30T06:26:02.865ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
110072025-04-30T06:26:02.866ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "031519fa-f82c-4928-923f-34014710d9c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110082025-04-30T06:26:02.867ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
110092025-04-30T06:26:02.868ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db820e45-67d3-4ad7-9f20-b4a3e30b672b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110102025-04-30T06:26:02.868ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
110112025-04-30T06:26:02.871ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5732d49-1659-443d-afe6-3dd7b18d492f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110122025-04-30T06:26:02.871ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
110132025-04-30T06:26:02.873ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20701159-f745-4ee8-9370-561363af9537", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110142025-04-30T06:26:02.873ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
110152025-04-30T06:26:02.875ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56dd9a09-2622-40d3-8e75-667f456b689e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110162025-04-30T06:26:02.875ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
110172025-04-30T06:26:02.878ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1039698-b530-46d9-8cb8-9b63f6d98553", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110182025-04-30T06:26:02.878ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
110192025-04-30T06:26:02.884ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77259943-0680-4387-af2f-9a00bd75f0be", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110202025-04-30T06:26:02.884ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
110212025-04-30T06:26:02.886ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b227d37b-5246-4621-9f7d-4433fee1e260", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110222025-04-30T06:26:02.886ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
110232025-04-30T06:26:02.894ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa89def-2b83-4b0b-a8c5-341ec8255412", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110242025-04-30T06:26:02.894ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
110252025-04-30T06:26:02.894ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4471af63-34ae-4a1b-b735-cfb5987b1fdb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110262025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
110272025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43f8084b-e9fa-4867-b1a3-8583c4a149e3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110282025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
110292025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebe6f175-ce24-47f9-8762-6be2a5ca8980", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110302025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
110312025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbbc34f7-2d10-49a6-b34d-4bb80b1cce04", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110322025-04-30T06:26:02.895ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
110332025-04-30T06:26:02.898ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1121adfe-0231-44da-9ba2-1ece2f97d63f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110342025-04-30T06:26:02.898ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
110352025-04-30T06:26:02.900ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110362025-04-30T06:26:02.900ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110372025-04-30T06:26:02.900ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110382025-04-30T06:26:02.900ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
110392025-04-30T06:26:02.902ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110402025-04-30T06:26:02.904ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110412025-04-30T06:26:02.915ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110422025-04-30T06:26:02.915ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110432025-04-30T06:26:02.915ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
110442025-04-30T06:26:02.917ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110452025-04-30T06:26:02.919ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d1c1083-8eff-46fa-b63d-ff2abe6108f7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110462025-04-30T06:26:02.919ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
110472025-04-30T06:26:02.920ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110482025-04-30T06:26:02.920ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
110492025-04-30T06:26:02.921ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110502025-04-30T06:26:02.925ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48bb899e-f2d8-4730-a00a-5e5e2f1b1460", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110512025-04-30T06:26:02.925ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
110522025-04-30T06:26:02.929ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b10975d-3292-467d-a3aa-c829e9eb2a29", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110532025-04-30T06:26:02.929ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
110542025-04-30T06:26:02.944ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bafef99-6dfc-4c48-a962-a3429a39be73", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110552025-04-30T06:26:02.944ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
110562025-04-30T06:26:02.948ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8801873-2f1a-48ca-8937-8a5729d343af", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110572025-04-30T06:26:02.948ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
110582025-04-30T06:26:02.949ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "612f3c59-77f7-4092-8f81-bb3c1ab6925e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110592025-04-30T06:26:02.949ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
110602025-04-30T06:26:02.953ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c34ccb5-026c-403e-bddf-d2b32cada2c2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110612025-04-30T06:26:02.953ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
110622025-04-30T06:26:02.954ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7380c753-4c34-4e2a-a56a-cfe0007e5864", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110632025-04-30T06:26:02.954ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
110642025-04-30T06:26:02.955ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b36c3a75-c091-49bc-93f6-5d278bacff55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110652025-04-30T06:26:02.955ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
110662025-04-30T06:26:02.955ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
110672025-04-30T06:26:02.958ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "89a43280-55f1-43c9-be73-03eef3a9dd68", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110682025-04-30T06:26:02.958ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
110692025-04-30T06:26:02.960ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "234996ef-7626-4b1c-8095-128c07f10cd7", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110702025-04-30T06:26:02.960ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
110712025-04-30T06:26:02.962ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b661fa92-b6f5-49e6-a963-865f1dda8987", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110722025-04-30T06:26:02.962ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
110732025-04-30T06:26:02.964ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0882bf3a-4a60-49a3-bac2-d0c3ae255d3a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110742025-04-30T06:26:02.964ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
110752025-04-30T06:26:02.969ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4f4aee8-c166-4438-bca5-66a82c959223", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
110762025-04-30T06:26:02.969ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
110772025-04-30T06:26:02.969ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110782025-04-30T06:26:02.969ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110792025-04-30T06:26:02.969ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110802025-04-30T06:26:02.972ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110812025-04-30T06:26:02.972ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
110822025-04-30T06:26:02.975ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110832025-04-30T06:26:02.981ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
110842025-04-30T06:26:02.981ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
110852025-04-30T06:26:02.981ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110862025-04-30T06:26:02.981ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110872025-04-30T06:26:02.981ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110882025-04-30T06:26:02.982ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110892025-04-30T06:26:02.985ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110902025-04-30T06:26:02.993ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
110912025-04-30T06:26:02.993ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
110922025-04-30T06:26:02.993ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
110932025-04-30T06:26:02.994ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
110942025-04-30T06:26:03.083ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
110952025-04-30T06:26:03.083ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
110962025-04-30T06:26:03.084ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110972025-04-30T06:26:03.084ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110982025-04-30T06:26:03.084ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110992025-04-30T06:26:03.084ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
111002025-04-30T06:26:03.085ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111012025-04-30T06:26:03.085ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:49484 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
111022025-04-30T06:26:03.085ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:49484 req_id = 264d7caa-5c38-4faf-a760-141d9c60a707 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
111032025-04-30T06:26:03.086ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 410 local_addr = [::1]:42668 method = GET remote_addr = [::1]:49484 req_id = 264d7caa-5c38-4faf-a760-141d9c60a707 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
111042025-04-30T06:26:03.086ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "264d7caa-5c38-4faf-a760-141d9c60a707", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
111052025-04-30T06:26:03.132ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
111062025-04-30T06:26:03.132ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
111072025-04-30T06:26:03.133ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111082025-04-30T06:26:03.138ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
111092025-04-30T06:26:03.138ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111102025-04-30T06:26:03.138ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
111112025-04-30T06:26:03.139ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111122025-04-30T06:26:03.139ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:63375 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
111132025-04-30T06:26:03.140ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:63375 req_id = 0d2c5c20-6518-4b5c-abe3-e116a137d0e1 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
111142025-04-30T06:26:03.140ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 342 local_addr = [::1]:47918 method = GET remote_addr = [::1]:63375 req_id = 0d2c5c20-6518-4b5c-abe3-e116a137d0e1 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
111152025-04-30T06:26:03.141ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d2c5c20-6518-4b5c-abe3-e116a137d0e1", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
111162025-04-30T06:26:03.199ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
111172025-04-30T06:26:03.200ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
111182025-04-30T06:26:03.200ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:50483 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
111192025-04-30T06:26:03.203ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
111202025-04-30T06:26:03.203ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
111212025-04-30T06:26:03.204ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111222025-04-30T06:26:03.211ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:50483 req_id = 208a31da-d85b-4725-a1a4-55b9fa918cef sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
111232025-04-30T06:26:03.212ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 498 local_addr = [::1]:35744 method = GET remote_addr = [::1]:50483 req_id = 208a31da-d85b-4725-a1a4-55b9fa918cef response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
111242025-04-30T06:26:03.214ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "208a31da-d85b-4725-a1a4-55b9fa918cef", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
111252025-04-30T06:26:03.289ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
111262025-04-30T06:26:03.289ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
111272025-04-30T06:26:03.299ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:39317 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111282025-04-30T06:26:03.301ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:39317 req_id = e270ddef-90d2-46f2-8133-f29fcf39873b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
111292025-04-30T06:26:03.301ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
111302025-04-30T06:26:03.307ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 6660 local_addr = [::1]:35343 method = GET remote_addr = [::1]:39317 req_id = e270ddef-90d2-46f2-8133-f29fcf39873b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
111312025-04-30T06:26:03.308ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111322025-04-30T06:26:03.309ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 72ab65fd-6dc4-46f0-952d-05106313bb42 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111332025-04-30T06:26:03.310ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e270ddef-90d2-46f2-8133-f29fcf39873b", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
111342025-04-30T06:26:03.312ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
111352025-04-30T06:26:03.312ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
111362025-04-30T06:26:03.312ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
111372025-04-30T06:26:03.319ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
111382025-04-30T06:26:03.319ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111392025-04-30T06:26:03.320ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111402025-04-30T06:26:03.320ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111412025-04-30T06:26:03.321ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111422025-04-30T06:26:03.322ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 72ab65fd-6dc4-46f0-952d-05106313bb42 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111432025-04-30T06:26:03.322ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 72ab65fd-6dc4-46f0-952d-05106313bb42 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111442025-04-30T06:26:03.324ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 72ab65fd-6dc4-46f0-952d-05106313bb42 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111452025-04-30T06:26:03.336ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:54020 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
111462025-04-30T06:26:03.337ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:54020 req_id = 584a3772-7d66-43e7-91f1-448c9916ace8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
111472025-04-30T06:26:03.345ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111482025-04-30T06:26:03.356ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
111492025-04-30T06:26:03.356ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111502025-04-30T06:26:03.357ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111512025-04-30T06:26:03.357ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111522025-04-30T06:26:03.358ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 21126 local_addr = [::1]:59920 method = GET remote_addr = [::1]:54020 req_id = 584a3772-7d66-43e7-91f1-448c9916ace8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
111532025-04-30T06:26:03.359ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 49357 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 72ab65fd-6dc4-46f0-952d-05106313bb42 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111542025-04-30T06:26:03.360ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "72ab65fd-6dc4-46f0-952d-05106313bb42", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
111552025-04-30T06:26:03.360ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
111562025-04-30T06:26:03.362ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 73ae9607-d742-4da8-bee3-3f160c260356 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
111572025-04-30T06:26:03.365ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 73ae9607-d742-4da8-bee3-3f160c260356 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
111582025-04-30T06:26:03.365ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 73ae9607-d742-4da8-bee3-3f160c260356 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
111592025-04-30T06:26:03.367ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 73ae9607-d742-4da8-bee3-3f160c260356 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
111602025-04-30T06:26:03.371ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 9606 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 73ae9607-d742-4da8-bee3-3f160c260356 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
111612025-04-30T06:26:03.374ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "73ae9607-d742-4da8-bee3-3f160c260356", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
111622025-04-30T06:26:03.375ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111632025-04-30T06:26:03.375ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
111642025-04-30T06:26:03.375ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
111652025-04-30T06:26:03.397ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
111662025-04-30T06:26:03.409ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111672025-04-30T06:26:03.409ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111682025-04-30T06:26:03.410ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111692025-04-30T06:26:03.411ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111702025-04-30T06:26:03.411ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111712025-04-30T06:26:03.411ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111722025-04-30T06:26:03.413ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
111732025-04-30T06:26:03.413ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
111742025-04-30T06:26:03.413ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 781.866197ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
111752025-04-30T06:26:03.413ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
111762025-04-30T06:26:03.413ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
111772025-04-30T06:26:03.413ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111782025-04-30T06:26:03.413ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111792025-04-30T06:26:03.413ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111802025-04-30T06:26:03.414ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111812025-04-30T06:26:03.416ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111822025-04-30T06:26:03.422ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
111832025-04-30T06:26:03.423ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
111842025-04-30T06:26:03.423ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
111852025-04-30T06:26:03.423ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
111862025-04-30T06:26:03.427ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
111872025-04-30T06:26:03.427ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
111882025-04-30T06:26:03.428ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 01 88 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62497 req_id = 86457524-6cb2-4edc-8817-cedce892d879
111892025-04-30T06:26:03.428ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 392,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62497 req_id = 86457524-6cb2-4edc-8817-cedce892d879
111902025-04-30T06:26:03.428ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
111912025-04-30T06:26:03.429ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
111922025-04-30T06:26:03.429ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111932025-04-30T06:26:03.429ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62497 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 86457524-6cb2-4edc-8817-cedce892d879
111942025-04-30T06:26:03.431ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3228, tv_nsec: 229971718 } })
111952025-04-30T06:26:03.431ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
111962025-04-30T06:26:03.431ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
111972025-04-30T06:26:03.438ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3e a0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e >...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:61593 req_id = c257f6ed-c969-4612-946e-79ea8fbc3ede
111982025-04-30T06:26:03.438ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16032,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61593 req_id = c257f6ed-c969-4612-946e-79ea8fbc3ede
111992025-04-30T06:26:03.438ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
112002025-04-30T06:26:03.438ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61593 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c257f6ed-c969-4612-946e-79ea8fbc3ede
112012025-04-30T06:26:03.438ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0e 67 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .g..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63679 req_id = 6462df7e-49df-4e5e-b8bd-5b9b396643c6
112022025-04-30T06:26:03.438ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3687,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63679 req_id = 6462df7e-49df-4e5e-b8bd-5b9b396643c6
112032025-04-30T06:26:03.438ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112042025-04-30T06:26:03.438ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63679 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6462df7e-49df-4e5e-b8bd-5b9b396643c6
112052025-04-30T06:26:03.552ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
112062025-04-30T06:26:03.552ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112072025-04-30T06:26:03.552ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112082025-04-30T06:26:03.552ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112092025-04-30T06:26:03.552ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
112102025-04-30T06:26:03.553ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112112025-04-30T06:26:03.559ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 5d d5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ]............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62222 req_id = 026a9333-6d7d-4f65-9a47-ee47a8c6c2c5
112122025-04-30T06:26:03.559ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24021,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62222 req_id = 026a9333-6d7d-4f65-9a47-ee47a8c6c2c5
112132025-04-30T06:26:03.559ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
112142025-04-30T06:26:03.560ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 24021, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62222 req_id = 026a9333-6d7d-4f65-9a47-ee47a8c6c2c5
112152025-04-30T06:26:03.560ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
112162025-04-30T06:26:03.560ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
112172025-04-30T06:26:03.560ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
112182025-04-30T06:26:03.566ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112192025-04-30T06:26:03.573ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112202025-04-30T06:26:03.573ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112212025-04-30T06:26:03.573ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
112222025-04-30T06:26:03.573ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112232025-04-30T06:26:03.580ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "749cfa31-736f-4c36-ab74-6c4a6b717107", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112242025-04-30T06:26:03.580ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
112252025-04-30T06:26:03.582ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112262025-04-30T06:26:03.582ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
112272025-04-30T06:26:03.583ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112282025-04-30T06:26:03.595ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99438194-9f79-49fc-8c06-ffa616b397f5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112292025-04-30T06:26:03.596ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
112302025-04-30T06:26:03.601ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a895762-061d-48f0-91e9-81fcea3c9ad5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112312025-04-30T06:26:03.601ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
112322025-04-30T06:26:03.606ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0038f150-b33a-4fb5-868e-b967089e7a73", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112332025-04-30T06:26:03.606ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
112342025-04-30T06:26:03.607ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
112352025-04-30T06:26:03.607ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
112362025-04-30T06:26:03.607ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
112372025-04-30T06:26:03.608ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
112382025-04-30T06:26:03.613ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
112392025-04-30T06:26:03.614ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
112402025-04-30T06:26:03.622ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
112412025-04-30T06:26:03.622ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
112422025-04-30T06:26:03.624ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "13f64df7-5f7c-4434-9547-7602216ffbba", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112432025-04-30T06:26:03.624ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
112442025-04-30T06:26:03.630ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6168c82-2f3d-40b2-84c5-869169511dd7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112452025-04-30T06:26:03.631ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
112462025-04-30T06:26:03.636ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
112472025-04-30T06:26:03.637ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1160536f-aa86-4b97-8f93-bdbb04a0d327", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112482025-04-30T06:26:03.637ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
112492025-04-30T06:26:03.645ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6135a04-0e07-4bf5-95dd-fafd16c2a46e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112502025-04-30T06:26:03.646ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
112512025-04-30T06:26:03.647ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "24bd09e5-12e0-4394-aea1-6a8624ff1255", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112522025-04-30T06:26:03.647ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
112532025-04-30T06:26:03.648ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8133b42e-cfc7-4b02-88cb-2eb9151690b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112542025-04-30T06:26:03.648ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
112552025-04-30T06:26:03.649ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e117e4c9-ac59-4ed6-9d4d-fc6397805e6a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112562025-04-30T06:26:03.649ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
112572025-04-30T06:26:03.665ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6941b621-d9b7-4908-8f5e-ce555115396b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112582025-04-30T06:26:03.665ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
112592025-04-30T06:26:03.667ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a54e5291-430b-4cd2-a7ea-63048d80f084", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112602025-04-30T06:26:03.667ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
112612025-04-30T06:26:03.670ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d23cc46-4ca6-420c-b4ae-d78d4a5bccd9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112622025-04-30T06:26:03.671ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
112632025-04-30T06:26:03.673ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "211a8196-aba0-4649-97d7-509bcd0bbdbb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112642025-04-30T06:26:03.673ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
112652025-04-30T06:26:03.677ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa08f900-99ba-461c-b192-3440ad54cdd2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112662025-04-30T06:26:03.678ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
112672025-04-30T06:26:03.680ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2be0d496-bb34-4b10-938c-0ce01fbe1d22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112682025-04-30T06:26:03.680ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
112692025-04-30T06:26:03.682ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "744fde93-491a-4a33-911a-ccbcd89e026f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112702025-04-30T06:26:03.682ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
112712025-04-30T06:26:03.685ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "059a9e0b-dd4a-4cc7-ba4b-32e8cd3154f3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112722025-04-30T06:26:03.685ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
112732025-04-30T06:26:03.687ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9df5475-c6e8-4912-987a-efaaaad0dd1f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112742025-04-30T06:26:03.687ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
112752025-04-30T06:26:03.687ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112762025-04-30T06:26:03.687ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112772025-04-30T06:26:03.687ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112782025-04-30T06:26:03.688ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
112792025-04-30T06:26:03.689ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112802025-04-30T06:26:03.690ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "927e0969-8a2f-4090-829d-a6609a55c2a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112812025-04-30T06:26:03.691ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
112822025-04-30T06:26:03.693ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112832025-04-30T06:26:03.704ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112842025-04-30T06:26:03.704ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112852025-04-30T06:26:03.704ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
112862025-04-30T06:26:03.705ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112872025-04-30T06:26:03.708ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab0ec7ae-5d70-4c26-a9ab-1fa3ce56927b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112882025-04-30T06:26:03.708ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
112892025-04-30T06:26:03.708ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
112902025-04-30T06:26:03.712ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112912025-04-30T06:26:03.712ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
112922025-04-30T06:26:03.712ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112932025-04-30T06:26:03.714ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb047046-6e80-4712-a00f-c59bbedc333c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112942025-04-30T06:26:03.714ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
112952025-04-30T06:26:03.718ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68ee99f5-b29a-40c1-9c94-bba46ed69a7f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112962025-04-30T06:26:03.719ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
112972025-04-30T06:26:03.723ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a98c794-fb07-4b12-b779-adfbb0c5dc0f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
112982025-04-30T06:26:03.724ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
112992025-04-30T06:26:03.727ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c9011b9-e7c3-43d9-8c92-1ef3096b82b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113002025-04-30T06:26:03.727ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
113012025-04-30T06:26:03.730ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04518b49-8218-47f9-8ad5-976e58e32e38", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113022025-04-30T06:26:03.731ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
113032025-04-30T06:26:03.735ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89a3487a-b283-4974-b831-b9504db1bfad", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113042025-04-30T06:26:03.735ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
113052025-04-30T06:26:03.739ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "14727762-9676-4798-9a64-a5c02c4fd72e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113062025-04-30T06:26:03.739ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
113072025-04-30T06:26:03.744ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cf3315d-1d59-407e-b878-0483e344fcca", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113082025-04-30T06:26:03.745ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
113092025-04-30T06:26:03.748ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70165dc8-5b58-4dec-8e5c-e836ab831111", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113102025-04-30T06:26:03.748ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
113112025-04-30T06:26:03.748ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
113122025-04-30T06:26:03.753ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07db4774-58ba-4531-b311-cea875b90310", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113132025-04-30T06:26:03.753ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
113142025-04-30T06:26:03.754ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48a3dd9e-68d5-4061-9bb4-6390a9c74a7a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113152025-04-30T06:26:03.755ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
113162025-04-30T06:26:03.756ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f012ec7f-abb6-4c95-b924-00c81e49d579", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113172025-04-30T06:26:03.756ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
113182025-04-30T06:26:03.757ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc7d9e68-290b-40cf-898d-6489b1f64068", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113192025-04-30T06:26:03.758ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
113202025-04-30T06:26:03.761ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d17dd11b-9edd-45dd-97cc-a40d4c7d0076", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113212025-04-30T06:26:03.761ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
113222025-04-30T06:26:03.764ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5778fa9b-2dcc-4abb-a2fd-699f025c403b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113232025-04-30T06:26:03.764ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
113242025-04-30T06:26:03.768ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8270f237-d098-4720-9ea3-bd0445d1414a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113252025-04-30T06:26:03.768ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
113262025-04-30T06:26:03.770ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9fb8c38f-4fe2-4ee7-87dc-513cef7fda8a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113272025-04-30T06:26:03.770ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
113282025-04-30T06:26:03.773ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bc2fd38-9686-4981-9093-f4bffa6e0e10", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113292025-04-30T06:26:03.783ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
113302025-04-30T06:26:03.785ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d1dd80a-fd1e-486b-ad4d-e2bfddd1e7ca", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113312025-04-30T06:26:03.786ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
113322025-04-30T06:26:03.786ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ceb8480-e1a3-4582-b9b6-9c1326cb6e4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113332025-04-30T06:26:03.786ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
113342025-04-30T06:26:03.789ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "530b23ff-7fdf-47b6-b500-b0ce6d10027e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113352025-04-30T06:26:03.789ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
113362025-04-30T06:26:03.790ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edf69a7d-2cfd-4449-a81f-a152c924cbf6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:02 GMT"} })
113372025-04-30T06:26:03.791ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
113382025-04-30T06:26:03.793ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "525c9c3c-225e-4b39-99aa-d115885107fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113392025-04-30T06:26:03.793ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
113402025-04-30T06:26:03.793ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
113412025-04-30T06:26:03.795ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "904d6bf3-290e-4abf-92b0-b5e93aeb459f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113422025-04-30T06:26:03.797ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
113432025-04-30T06:26:03.797ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d7db643-625b-4147-89b0-15cace25839b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113442025-04-30T06:26:03.798ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
113452025-04-30T06:26:03.799ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac1cabaf-7d85-476c-a237-e6765402d4c2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113462025-04-30T06:26:03.800ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
113472025-04-30T06:26:03.800ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113482025-04-30T06:26:03.800ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113492025-04-30T06:26:03.800ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113502025-04-30T06:26:03.800ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
113512025-04-30T06:26:03.801ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113522025-04-30T06:26:03.802ZDEBGtest_replacement_sanity (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
113532025-04-30T06:26:03.802ZWARNtest_replacement_sanity (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
113542025-04-30T06:26:03.804ZWARNtest_replacement_sanity (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
113552025-04-30T06:26:03.805ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c9eaa04-1563-418c-a093-21924707542d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113562025-04-30T06:26:03.805ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
113572025-04-30T06:26:03.806ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113582025-04-30T06:26:03.814ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113592025-04-30T06:26:03.814ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113602025-04-30T06:26:03.815ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
113612025-04-30T06:26:03.816ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113622025-04-30T06:26:03.819ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113632025-04-30T06:26:03.819ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
113642025-04-30T06:26:03.820ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113652025-04-30T06:26:03.821ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10c5232f-f459-49df-b922-2b4cc3918e76", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113662025-04-30T06:26:03.821ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113672025-04-30T06:26:03.821ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113682025-04-30T06:26:03.821ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113692025-04-30T06:26:03.821ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113702025-04-30T06:26:03.822ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113712025-04-30T06:26:03.826ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113722025-04-30T06:26:03.832ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113732025-04-30T06:26:03.833ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113742025-04-30T06:26:03.833ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113752025-04-30T06:26:03.833ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113762025-04-30T06:26:03.833ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113772025-04-30T06:26:03.834ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113782025-04-30T06:26:03.836ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113792025-04-30T06:26:03.843ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113802025-04-30T06:26:03.843ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113812025-04-30T06:26:03.843ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113822025-04-30T06:26:03.844ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113832025-04-30T06:26:03.909ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
113842025-04-30T06:26:03.909ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
113852025-04-30T06:26:03.948ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:37315 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
113862025-04-30T06:26:03.952ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:37315 req_id = a0c218ef-5c1d-4ee5-af9f-e4b92ee1defa sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
113872025-04-30T06:26:04.617ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 664549 local_addr = [::1]:42668 method = GET remote_addr = [::1]:37315 req_id = a0c218ef-5c1d-4ee5-af9f-e4b92ee1defa response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
113882025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:52095 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113892025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:35689 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113902025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:58039 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113912025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:43057 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113922025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:42805 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113932025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:42375 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113942025-04-30T06:26:04.618ZINFOtest_replacement_sanity (dropshot): accepted connection kind = pantry local_addr = [::1]:59920 remote_addr = [::1]:42058 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113952025-04-30T06:26:04.619ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
113962025-04-30T06:26:04.619ZTRACtest_replacement_sanity (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113972025-04-30T06:26:04.619ZTRACtest_replacement_sanity (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
113982025-04-30T06:26:04.619ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0c218ef-5c1d-4ee5-af9f-e4b92ee1defa", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
113992025-04-30T06:26:04.693ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
114002025-04-30T06:26:04.693ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
114012025-04-30T06:26:04.697ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114022025-04-30T06:26:04.701ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:50121 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
114032025-04-30T06:26:04.702ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:52095 req_id = c957a2a4-929e-4fc8-ae9d-b059d3ba9d05 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114042025-04-30T06:26:04.702ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:35689 req_id = e3cb2a8d-1b03-47cc-8057-7b29350d5a3d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114052025-04-30T06:26:04.702ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:58039 req_id = c6a5db15-71f5-4f6d-9300-d5ad568a91d9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114062025-04-30T06:26:04.703ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:43057 req_id = 0e6571e0-d0cb-41d5-81d7-d49f46ebf2db sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114072025-04-30T06:26:04.703ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:42805 req_id = 8d398148-1339-424b-95e9-732b74ff716b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114082025-04-30T06:26:04.703ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:42375 req_id = 4c8d2e98-b570-4ecd-a557-a4b5694c24d1 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114092025-04-30T06:26:04.703ZTRACtest_replacement_sanity (dropshot): incoming request kind = pantry local_addr = [::1]:59920 method = GET remote_addr = [::1]:42058 req_id = 6d248c60-3c28-4a7f-8282-53a7e91f0242 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114102025-04-30T06:26:04.705ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3686 local_addr = [::1]:59920 method = GET remote_addr = [::1]:52095 req_id = c957a2a4-929e-4fc8-ae9d-b059d3ba9d05 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114112025-04-30T06:26:04.706ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3663 local_addr = [::1]:59920 method = GET remote_addr = [::1]:35689 req_id = e3cb2a8d-1b03-47cc-8057-7b29350d5a3d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114122025-04-30T06:26:04.706ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3467 local_addr = [::1]:59920 method = GET remote_addr = [::1]:58039 req_id = c6a5db15-71f5-4f6d-9300-d5ad568a91d9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114132025-04-30T06:26:04.706ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3420 local_addr = [::1]:59920 method = GET remote_addr = [::1]:43057 req_id = 0e6571e0-d0cb-41d5-81d7-d49f46ebf2db response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114142025-04-30T06:26:04.706ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3322 local_addr = [::1]:59920 method = GET remote_addr = [::1]:42805 req_id = 8d398148-1339-424b-95e9-732b74ff716b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114152025-04-30T06:26:04.706ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3281 local_addr = [::1]:59920 method = GET remote_addr = [::1]:42375 req_id = 4c8d2e98-b570-4ecd-a557-a4b5694c24d1 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114162025-04-30T06:26:04.707ZINFOtest_replacement_sanity (dropshot): request completed kind = pantry latency_us = 3300 local_addr = [::1]:59920 method = GET remote_addr = [::1]:42058 req_id = 6d248c60-3c28-4a7f-8282-53a7e91f0242 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
114172025-04-30T06:26:04.709ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:50121 req_id = 108f3d12-b49b-488e-aff6-b28436fc8d3c sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
114182025-04-30T06:26:04.711ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114192025-04-30T06:26:04.712ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114202025-04-30T06:26:04.713ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114212025-04-30T06:26:04.713ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114222025-04-30T06:26:04.714ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 5212 local_addr = [::1]:47918 method = GET remote_addr = [::1]:50121 req_id = 108f3d12-b49b-488e-aff6-b28436fc8d3c response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
114232025-04-30T06:26:04.733ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "108f3d12-b49b-488e-aff6-b28436fc8d3c", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:03 GMT"} })
114242025-04-30T06:26:04.845ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
114252025-04-30T06:26:04.846ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
114262025-04-30T06:26:04.847ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:59110 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
114272025-04-30T06:26:04.860ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:59110 req_id = 05ba6bb4-0b07-4d16-ae81-377b7a78d606 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
114282025-04-30T06:26:04.860ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 347 local_addr = [::1]:35744 method = GET remote_addr = [::1]:59110 req_id = 05ba6bb4-0b07-4d16-ae81-377b7a78d606 response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
114292025-04-30T06:26:04.883ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "05ba6bb4-0b07-4d16-ae81-377b7a78d606", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
114302025-04-30T06:26:04.966ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
114312025-04-30T06:26:04.966ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
114322025-04-30T06:26:04.967ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
114332025-04-30T06:26:04.969ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:63195 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
114342025-04-30T06:26:04.973ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:63195 req_id = 5c7154ee-6399-4b51-b20f-eb6342940783 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114352025-04-30T06:26:04.982ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 8680 local_addr = [::1]:35343 method = GET remote_addr = [::1]:63195 req_id = 5c7154ee-6399-4b51-b20f-eb6342940783 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
114362025-04-30T06:26:04.984ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c7154ee-6399-4b51-b20f-eb6342940783", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
114372025-04-30T06:26:04.986ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114382025-04-30T06:26:04.986ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
114392025-04-30T06:26:04.986ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
114402025-04-30T06:26:04.993ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114412025-04-30T06:26:04.993ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114422025-04-30T06:26:04.993ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114432025-04-30T06:26:04.993ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114442025-04-30T06:26:04.994ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114452025-04-30T06:26:05.012ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114462025-04-30T06:26:05.021ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114472025-04-30T06:26:05.022ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114482025-04-30T06:26:05.022ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114492025-04-30T06:26:05.023ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114502025-04-30T06:26:05.024ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
114512025-04-30T06:26:05.024ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114522025-04-30T06:26:05.024ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114532025-04-30T06:26:05.024ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
114542025-04-30T06:26:05.024ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114552025-04-30T06:26:05.057ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114562025-04-30T06:26:05.069ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114572025-04-30T06:26:05.069ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114582025-04-30T06:26:05.070ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
114592025-04-30T06:26:05.070ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114602025-04-30T06:26:05.116ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
114612025-04-30T06:26:05.118ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
114622025-04-30T06:26:05.118ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.705156031s iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
114632025-04-30T06:26:05.118ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
114642025-04-30T06:26:05.118ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
114652025-04-30T06:26:05.119ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
114662025-04-30T06:26:05.119ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114672025-04-30T06:26:05.119ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114682025-04-30T06:26:05.120ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114692025-04-30T06:26:05.125ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114702025-04-30T06:26:05.125ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
114712025-04-30T06:26:05.126ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114722025-04-30T06:26:05.130ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114732025-04-30T06:26:05.137ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
114742025-04-30T06:26:05.137ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
114752025-04-30T06:26:05.137ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
114762025-04-30T06:26:05.138ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
114772025-04-30T06:26:05.141ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
114782025-04-30T06:26:05.141ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
114792025-04-30T06:26:05.144ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b4 09 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53129 req_id = 84b986e8-7171-496c-8038-c6c590ccd89e
114802025-04-30T06:26:05.144ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46089,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53129 req_id = 84b986e8-7171-496c-8038-c6c590ccd89e
114812025-04-30T06:26:05.144ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
114822025-04-30T06:26:05.145ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
114832025-04-30T06:26:05.146ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
114842025-04-30T06:26:05.146ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53129 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 84b986e8-7171-496c-8038-c6c590ccd89e
114852025-04-30T06:26:05.147ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3229, tv_nsec: 946639216 } })
114862025-04-30T06:26:05.147ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
114872025-04-30T06:26:05.147ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
114882025-04-30T06:26:05.160ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 69 db 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e i...........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58124 req_id = a811855e-8c8a-44e6-8b59-0ab445e20497
114892025-04-30T06:26:05.161ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27099,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58124 req_id = a811855e-8c8a-44e6-8b59-0ab445e20497
114902025-04-30T06:26:05.161ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
114912025-04-30T06:26:05.161ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58124 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a811855e-8c8a-44e6-8b59-0ab445e20497
114922025-04-30T06:26:05.161ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 95 fc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57733 req_id = 06e5f0bc-e734-4acd-9525-c861048a942b
114932025-04-30T06:26:05.162ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38396,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57733 req_id = 06e5f0bc-e734-4acd-9525-c861048a942b
114942025-04-30T06:26:05.162ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
114952025-04-30T06:26:05.162ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57733 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 06e5f0bc-e734-4acd-9525-c861048a942b
114962025-04-30T06:26:05.301ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
114972025-04-30T06:26:05.304ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
114982025-04-30T06:26:05.313ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
114992025-04-30T06:26:05.314ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 777b9805-18a1-4f48-801e-eb662596ee74 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115002025-04-30T06:26:05.314ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d6 d9 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62297 req_id = 50df6cc7-d702-4634-b123-b5f5c0d0daa7
115012025-04-30T06:26:05.315ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55001,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62297 req_id = 50df6cc7-d702-4634-b123-b5f5c0d0daa7
115022025-04-30T06:26:05.315ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
115032025-04-30T06:26:05.315ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55001, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62297 req_id = 50df6cc7-d702-4634-b123-b5f5c0d0daa7
115042025-04-30T06:26:05.316ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 777b9805-18a1-4f48-801e-eb662596ee74 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115052025-04-30T06:26:05.316ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 777b9805-18a1-4f48-801e-eb662596ee74 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115062025-04-30T06:26:05.317ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 777b9805-18a1-4f48-801e-eb662596ee74 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115072025-04-30T06:26:05.318ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
115082025-04-30T06:26:05.318ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
115092025-04-30T06:26:05.318ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
115102025-04-30T06:26:05.330ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 16779 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 777b9805-18a1-4f48-801e-eb662596ee74 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
115112025-04-30T06:26:05.333ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "777b9805-18a1-4f48-801e-eb662596ee74", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115122025-04-30T06:26:05.333ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
115132025-04-30T06:26:05.334ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = cf2fc7b7-d1a2-4922-bd03-41fbeb1d4079 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
115142025-04-30T06:26:05.336ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = cf2fc7b7-d1a2-4922-bd03-41fbeb1d4079 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
115152025-04-30T06:26:05.336ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = cf2fc7b7-d1a2-4922-bd03-41fbeb1d4079 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
115162025-04-30T06:26:05.337ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = cf2fc7b7-d1a2-4922-bd03-41fbeb1d4079 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
115172025-04-30T06:26:05.339ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "888431fc-7ba5-4214-95e0-d05fe53ae9ee", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115182025-04-30T06:26:05.339ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
115192025-04-30T06:26:05.342ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 8402 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = cf2fc7b7-d1a2-4922-bd03-41fbeb1d4079 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
115202025-04-30T06:26:05.343ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf2fc7b7-d1a2-4922-bd03-41fbeb1d4079", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115212025-04-30T06:26:05.343ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115222025-04-30T06:26:05.344ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
115232025-04-30T06:26:05.344ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
115242025-04-30T06:26:05.359ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5014df4-fe39-416b-b700-9734e122e010", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115252025-04-30T06:26:05.361ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
115262025-04-30T06:26:05.362ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
115272025-04-30T06:26:05.367ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c5eeb1d-b83e-4e39-907b-b83e08a4e79f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115282025-04-30T06:26:05.367ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
115292025-04-30T06:26:05.372ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "46460d06-b27b-4c2f-80a0-bd292c9c44ba", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115302025-04-30T06:26:05.372ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
115312025-04-30T06:26:05.376ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79f3bd0e-e0a1-48c1-b800-0d8dd8b4007b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115322025-04-30T06:26:05.376ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
115332025-04-30T06:26:05.382ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a51ec984-07dc-4273-8a9c-cd61944cbe77", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115342025-04-30T06:26:05.382ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
115352025-04-30T06:26:05.385ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b59a55e-e097-49b0-9f7d-4528119c5475", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115362025-04-30T06:26:05.391ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
115372025-04-30T06:26:05.395ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21be0e9a-794d-45b1-b821-8797cfff8fe1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115382025-04-30T06:26:05.395ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
115392025-04-30T06:26:05.396ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e79a5f16-a32d-42c6-829b-ed4e221b877f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115402025-04-30T06:26:05.397ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
115412025-04-30T06:26:05.399ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52028603-3d30-4989-9974-1cdddcd2a251", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115422025-04-30T06:26:05.399ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
115432025-04-30T06:26:05.400ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "372493a9-574d-49a8-a54c-ff5dab233fc5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115442025-04-30T06:26:05.401ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
115452025-04-30T06:26:05.402ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09646a8e-a3f7-48e8-b11b-b8ce267a7fe6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115462025-04-30T06:26:05.402ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
115472025-04-30T06:26:05.405ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f155dc1-864e-4492-924e-44568c179bd4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115482025-04-30T06:26:05.405ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
115492025-04-30T06:26:05.408ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115502025-04-30T06:26:05.408ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115512025-04-30T06:26:05.410ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115522025-04-30T06:26:05.411ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115532025-04-30T06:26:05.411ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115542025-04-30T06:26:05.411ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
115552025-04-30T06:26:05.413ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8996f7b-70e7-44d1-b28f-372f3007a28d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115562025-04-30T06:26:05.413ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
115572025-04-30T06:26:05.413ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115582025-04-30T06:26:05.414ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115592025-04-30T06:26:05.414ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115602025-04-30T06:26:05.414ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
115612025-04-30T06:26:05.415ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115622025-04-30T06:26:05.419ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115632025-04-30T06:26:05.424ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115642025-04-30T06:26:05.424ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115652025-04-30T06:26:05.424ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
115662025-04-30T06:26:05.425ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115672025-04-30T06:26:05.428ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c760b43-3c3b-40d0-ac21-0344fafce679", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115682025-04-30T06:26:05.428ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
115692025-04-30T06:26:05.428ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115702025-04-30T06:26:05.428ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
115712025-04-30T06:26:05.429ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115722025-04-30T06:26:05.487ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99398fca-ced2-4d00-adb4-ed6845a1c304", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115732025-04-30T06:26:05.488ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
115742025-04-30T06:26:05.496ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "90921833-19e1-4bef-b403-d7b73eb943a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115752025-04-30T06:26:05.496ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
115762025-04-30T06:26:05.501ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
115772025-04-30T06:26:05.503ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "485d3bc4-c0b7-49ff-b50a-d5c4bb9d45fc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115782025-04-30T06:26:05.503ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
115792025-04-30T06:26:05.508ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a9c863e-579b-4310-b9ff-12fb3511d8e3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115802025-04-30T06:26:05.508ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
115812025-04-30T06:26:05.512ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5a119a8-4ddc-4048-9fc6-30fdd8fa1bce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115822025-04-30T06:26:05.512ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
115832025-04-30T06:26:05.514ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46f25119-2433-479b-bc86-f190811a91a5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115842025-04-30T06:26:05.514ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
115852025-04-30T06:26:05.516ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db6cd375-2ede-49c3-81b0-f368b47c2a15", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115862025-04-30T06:26:05.516ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
115872025-04-30T06:26:05.517ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fa79c67-8d57-48e3-8a81-793c4063eb9b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115882025-04-30T06:26:05.518ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
115892025-04-30T06:26:05.520ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aca9cf4b-6413-4cfa-b682-a604bdadeb2e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115902025-04-30T06:26:05.520ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
115912025-04-30T06:26:05.528ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c1d9205-2fef-4fbd-b414-451a13f55bdb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115922025-04-30T06:26:05.528ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
115932025-04-30T06:26:05.533ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d08e497-3804-47de-846d-5e5a7abc0eb6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115942025-04-30T06:26:05.534ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
115952025-04-30T06:26:05.538ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0eb798e-6d6b-4546-9ac9-7500554aaf3d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115962025-04-30T06:26:05.539ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
115972025-04-30T06:26:05.543ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c81a5b-7032-482b-a577-daf48733f8d2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
115982025-04-30T06:26:05.544ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
115992025-04-30T06:26:05.548ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f12d138a-884d-4d6d-af9f-c45d361098e7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116002025-04-30T06:26:05.548ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
116012025-04-30T06:26:05.551ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4a2550c-804a-4750-912e-46a6efaf1580", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116022025-04-30T06:26:05.552ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
116032025-04-30T06:26:05.552ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116042025-04-30T06:26:05.552ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116052025-04-30T06:26:05.552ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116062025-04-30T06:26:05.552ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
116072025-04-30T06:26:05.553ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116082025-04-30T06:26:05.556ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ff597b3-0e2a-4c0a-923e-f0a0a6ba0683", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116092025-04-30T06:26:05.556ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
116102025-04-30T06:26:05.557ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116112025-04-30T06:26:05.561ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116122025-04-30T06:26:05.561ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116132025-04-30T06:26:05.561ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
116142025-04-30T06:26:05.562ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116152025-04-30T06:26:05.565ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0109513-b7b8-48df-b94b-c082776116a5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116162025-04-30T06:26:05.565ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
116172025-04-30T06:26:05.566ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116182025-04-30T06:26:05.566ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
116192025-04-30T06:26:05.566ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116202025-04-30T06:26:05.568ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c041503-740e-441f-aacc-a8ec97758a9f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116212025-04-30T06:26:05.568ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
116222025-04-30T06:26:05.570ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "437c2947-5521-4f38-9fbc-4f2d5689359f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116232025-04-30T06:26:05.570ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
116242025-04-30T06:26:05.573ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43fa0d19-6bd8-4f97-bd99-a9e58913563d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116252025-04-30T06:26:05.573ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
116262025-04-30T06:26:05.577ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca61dabd-1af0-4ac1-b3d5-8fe6d0f1e4f1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116272025-04-30T06:26:05.578ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
116282025-04-30T06:26:05.582ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfaca787-f52d-4c2b-a276-f82ddc3425bb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116292025-04-30T06:26:05.583ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
116302025-04-30T06:26:05.588ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c403683c-62d6-4b14-b53d-75d87478c92a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116312025-04-30T06:26:05.588ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
116322025-04-30T06:26:05.594ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2debfb61-b4fc-409e-837a-865559e86e8f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116332025-04-30T06:26:05.594ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
116342025-04-30T06:26:05.598ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
116352025-04-30T06:26:05.601ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f820fd46-a3a3-442b-a4fb-a87346c09660", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116362025-04-30T06:26:05.601ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
116372025-04-30T06:26:05.605ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8edcfcfb-c20a-4e0e-9641-ac314f61f021", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116382025-04-30T06:26:05.605ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
116392025-04-30T06:26:05.607ZDEBGtest_replacement_sanity (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
116402025-04-30T06:26:05.607ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
116412025-04-30T06:26:05.607ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd6a078b-a58b-4eec-bf4e-23a9a0288f22", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116422025-04-30T06:26:05.607ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
116432025-04-30T06:26:05.608ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
116442025-04-30T06:26:05.609ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
116452025-04-30T06:26:05.612ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb634ef4-e9af-45d7-8428-75136a024f12", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116462025-04-30T06:26:05.612ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
116472025-04-30T06:26:05.614ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8bf030b-f0c2-44d1-a789-6335a04d8157", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116482025-04-30T06:26:05.614ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
116492025-04-30T06:26:05.615ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
116502025-04-30T06:26:05.616ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
116512025-04-30T06:26:05.616ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "babc930c-5b67-4b86-80b2-a34587e574d0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116522025-04-30T06:26:05.616ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
116532025-04-30T06:26:05.616ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
116542025-04-30T06:26:05.620ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "874873ed-7489-44cb-a16a-dd420869ad64", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116552025-04-30T06:26:05.620ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
116562025-04-30T06:26:05.623ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = measurements_i64
116572025-04-30T06:26:05.623ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
116582025-04-30T06:26:05.625ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ac99b82-5cb9-45a0-8557-7dc357cb68a8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116592025-04-30T06:26:05.625ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
116602025-04-30T06:26:05.626ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbaddc92-4b95-4b5e-9fb8-703041866dd6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116612025-04-30T06:26:05.626ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
116622025-04-30T06:26:05.629ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ddaed8e0-e36d-4320-8fdf-10a63ab8aac8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116632025-04-30T06:26:05.629ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
116642025-04-30T06:26:05.632ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d45eca5f-3683-42f2-987f-913ff974c51e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
116652025-04-30T06:26:05.632ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
116662025-04-30T06:26:05.632ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116672025-04-30T06:26:05.632ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116682025-04-30T06:26:05.632ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116692025-04-30T06:26:05.635ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116702025-04-30T06:26:05.639ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116712025-04-30T06:26:05.645ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
116722025-04-30T06:26:05.646ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
116732025-04-30T06:26:05.646ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116742025-04-30T06:26:05.646ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116752025-04-30T06:26:05.646ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116762025-04-30T06:26:05.647ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116772025-04-30T06:26:05.651ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116782025-04-30T06:26:05.651ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116792025-04-30T06:26:05.651ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116802025-04-30T06:26:05.651ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
116812025-04-30T06:26:05.652ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116822025-04-30T06:26:05.654ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116832025-04-30T06:26:05.662ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
116842025-04-30T06:26:05.662ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
116852025-04-30T06:26:05.662ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
116862025-04-30T06:26:05.664ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116872025-04-30T06:26:05.666ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116882025-04-30T06:26:05.675ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116892025-04-30T06:26:05.675ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116902025-04-30T06:26:05.675ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
116912025-04-30T06:26:05.676ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116922025-04-30T06:26:05.681ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116932025-04-30T06:26:05.681ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
116942025-04-30T06:26:05.682ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116952025-04-30T06:26:05.746ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
116962025-04-30T06:26:05.746ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
116972025-04-30T06:26:05.746ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:60740 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
116982025-04-30T06:26:05.747ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
116992025-04-30T06:26:05.749ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:60740 req_id = ff3e36ae-084d-4f72-b04d-33a370cd4bc4 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
117002025-04-30T06:26:05.749ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 679 local_addr = [::1]:42668 method = GET remote_addr = [::1]:60740 req_id = ff3e36ae-084d-4f72-b04d-33a370cd4bc4 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
117012025-04-30T06:26:05.750ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff3e36ae-084d-4f72-b04d-33a370cd4bc4", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
117022025-04-30T06:26:05.820ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
117032025-04-30T06:26:05.820ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
117042025-04-30T06:26:05.821ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:51243 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
117052025-04-30T06:26:05.822ZTRACtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
117062025-04-30T06:26:05.823ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:51243 req_id = 93063f7d-4bb6-4f1b-95d1-f8658d452a6e sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
117072025-04-30T06:26:05.823ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 494 local_addr = [::1]:47918 method = GET remote_addr = [::1]:51243 req_id = 93063f7d-4bb6-4f1b-95d1-f8658d452a6e response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
117082025-04-30T06:26:05.824ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 76 14 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 v............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52583 req_id = 33608109-d272-4df4-ac96-4282bc791de0
117092025-04-30T06:26:05.825ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30228,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52583 req_id = 33608109-d272-4df4-ac96-4282bc791de0
117102025-04-30T06:26:05.825ZDEBGtest_replacement_sanity (store): query key key = _dendrite._tcp
117112025-04-30T06:26:05.825ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
117122025-04-30T06:26:05.825ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117132025-04-30T06:26:05.825ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52583 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 33608109-d272-4df4-ac96-4282bc791de0
117142025-04-30T06:26:05.827ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "93063f7d-4bb6-4f1b-95d1-f8658d452a6e", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:04 GMT"} })
117152025-04-30T06:26:05.891ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
117162025-04-30T06:26:05.891ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
117172025-04-30T06:26:05.891ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:51281 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
117182025-04-30T06:26:05.892ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 32936, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 53480, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3230, tv_nsec: 691332979 } })
117192025-04-30T06:26:05.892ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
117202025-04-30T06:26:05.892ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
117212025-04-30T06:26:05.894ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:51281 req_id = 6410e3c8-5af7-4cbd-851d-828ea0eca85a sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
117222025-04-30T06:26:05.894ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 286 local_addr = [::1]:35744 method = GET remote_addr = [::1]:51281 req_id = 6410e3c8-5af7-4cbd-851d-828ea0eca85a response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
117232025-04-30T06:26:05.894ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b6 cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58176 req_id = a3883c84-db81-4aae-b671-458953ab486b
117242025-04-30T06:26:05.894ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46795,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58176 req_id = a3883c84-db81-4aae-b671-458953ab486b
117252025-04-30T06:26:05.895ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
117262025-04-30T06:26:05.895ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58176 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a3883c84-db81-4aae-b671-458953ab486b
117272025-04-30T06:26:05.895ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e7 66 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .f..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52337 req_id = 1aebbda1-73cc-4d38-9ac7-3fae6dc46e43
117282025-04-30T06:26:05.895ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59238,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52337 req_id = 1aebbda1-73cc-4d38-9ac7-3fae6dc46e43
117292025-04-30T06:26:05.895ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117302025-04-30T06:26:05.895ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52337 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1aebbda1-73cc-4d38-9ac7-3fae6dc46e43
117312025-04-30T06:26:05.895ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6410e3c8-5af7-4cbd-851d-828ea0eca85a", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
117322025-04-30T06:26:05.951ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
117332025-04-30T06:26:05.951ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
117342025-04-30T06:26:05.951ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:42208 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
117352025-04-30T06:26:05.952ZINFOtest_replacement_sanity (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
117362025-04-30T06:26:06.035ZINFOtest_replacement_sanity (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
117372025-04-30T06:26:06.035ZDEBGtest_replacement_sanity (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
117382025-04-30T06:26:06.037ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:42208 req_id = d34813d7-4354-4b0e-b3d4-47545e9167e8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117392025-04-30T06:26:06.040ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 3650 local_addr = [::1]:35343 method = GET remote_addr = [::1]:42208 req_id = d34813d7-4354-4b0e-b3d4-47545e9167e8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117402025-04-30T06:26:06.042ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d34813d7-4354-4b0e-b3d4-47545e9167e8", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
117412025-04-30T06:26:06.052ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117422025-04-30T06:26:06.052ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117432025-04-30T06:26:06.052ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
117442025-04-30T06:26:06.062ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117452025-04-30T06:26:06.063ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117462025-04-30T06:26:06.063ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117472025-04-30T06:26:06.063ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117482025-04-30T06:26:06.065ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117492025-04-30T06:26:06.070ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117502025-04-30T06:26:06.075ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117512025-04-30T06:26:06.075ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117522025-04-30T06:26:06.075ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117532025-04-30T06:26:06.076ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117542025-04-30T06:26:06.090ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
117552025-04-30T06:26:06.111ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
117562025-04-30T06:26:06.111ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
117572025-04-30T06:26:06.111ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 992.51474ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
117582025-04-30T06:26:06.111ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
117592025-04-30T06:26:06.111ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117602025-04-30T06:26:06.111ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117612025-04-30T06:26:06.111ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117622025-04-30T06:26:06.111ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117632025-04-30T06:26:06.112ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117642025-04-30T06:26:06.115ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117652025-04-30T06:26:06.121ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117662025-04-30T06:26:06.121ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117672025-04-30T06:26:06.121ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117682025-04-30T06:26:06.122ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117692025-04-30T06:26:06.126ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
117702025-04-30T06:26:06.126ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
117712025-04-30T06:26:06.127ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6d 15 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 m............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56572 req_id = e37a77da-71f4-4254-af82-3d87015aafca
117722025-04-30T06:26:06.127ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27925,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56572 req_id = e37a77da-71f4-4254-af82-3d87015aafca
117732025-04-30T06:26:06.127ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
117742025-04-30T06:26:06.127ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
117752025-04-30T06:26:06.128ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117762025-04-30T06:26:06.128ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56572 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e37a77da-71f4-4254-af82-3d87015aafca
117772025-04-30T06:26:06.128ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3230, tv_nsec: 927564896 } })
117782025-04-30T06:26:06.128ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
117792025-04-30T06:26:06.128ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
117802025-04-30T06:26:06.129ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 86 66 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .f..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53889 req_id = 331ba669-96bd-4fd0-b312-ec3476a585b3
117812025-04-30T06:26:06.129ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34406,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53889 req_id = 331ba669-96bd-4fd0-b312-ec3476a585b3
117822025-04-30T06:26:06.129ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
117832025-04-30T06:26:06.129ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53889 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 331ba669-96bd-4fd0-b312-ec3476a585b3
117842025-04-30T06:26:06.132ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e5 f7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52921 req_id = 902a73fe-c176-4389-9c6d-22d946f0031b
117852025-04-30T06:26:06.132ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58871,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52921 req_id = 902a73fe-c176-4389-9c6d-22d946f0031b
117862025-04-30T06:26:06.132ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
117872025-04-30T06:26:06.132ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52921 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 902a73fe-c176-4389-9c6d-22d946f0031b
117882025-04-30T06:26:06.237ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
117892025-04-30T06:26:06.238ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117902025-04-30T06:26:06.238ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117912025-04-30T06:26:06.238ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117922025-04-30T06:26:06.238ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
117932025-04-30T06:26:06.240ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117942025-04-30T06:26:06.243ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f4 f6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54827 req_id = b71e72bf-1152-4e09-868f-15d29c40a39f
117952025-04-30T06:26:06.243ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62710,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54827 req_id = b71e72bf-1152-4e09-868f-15d29c40a39f
117962025-04-30T06:26:06.243ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
117972025-04-30T06:26:06.244ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62710, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54827 req_id = b71e72bf-1152-4e09-868f-15d29c40a39f
117982025-04-30T06:26:06.247ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
117992025-04-30T06:26:06.247ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
118002025-04-30T06:26:06.247ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
118012025-04-30T06:26:06.248ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118022025-04-30T06:26:06.261ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118032025-04-30T06:26:06.261ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118042025-04-30T06:26:06.261ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
118052025-04-30T06:26:06.262ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118062025-04-30T06:26:06.274ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118072025-04-30T06:26:06.275ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
118082025-04-30T06:26:06.275ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118092025-04-30T06:26:06.292ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd8026d8-a04a-4cc2-b2e7-c64ebababe50", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118102025-04-30T06:26:06.292ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
118112025-04-30T06:26:06.314ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db55ea8c-7610-4f3d-b894-302aa5555f99", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118122025-04-30T06:26:06.314ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
118132025-04-30T06:26:06.332ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7d64513-888a-47f5-bdf6-82a2d88d95a0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118142025-04-30T06:26:06.332ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
118152025-04-30T06:26:06.335ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ffcb103-d607-4e07-995d-3001ae93d488", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118162025-04-30T06:26:06.335ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
118172025-04-30T06:26:06.340ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d2b26b4-7e81-4392-8776-4021a446911d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118182025-04-30T06:26:06.340ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
118192025-04-30T06:26:06.345ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8db9a42-8654-426f-92fb-353aa997e708", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118202025-04-30T06:26:06.345ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
118212025-04-30T06:26:06.346ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
118222025-04-30T06:26:06.351ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d176807c-681b-4609-b802-9ced478ec180", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118232025-04-30T06:26:06.351ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
118242025-04-30T06:26:06.353ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a97526d-93ac-47f3-bff7-b1dee9bc1f47", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118252025-04-30T06:26:06.353ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
118262025-04-30T06:26:06.356ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04e3dcba-4271-4602-837b-ee4356ff4bbe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118272025-04-30T06:26:06.357ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
118282025-04-30T06:26:06.358ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5552aa41-4f69-48f7-ba07-2f4e4fb69478", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118292025-04-30T06:26:06.358ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
118302025-04-30T06:26:06.359ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eef7eab-080f-4208-8b04-91cdf298e929", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118312025-04-30T06:26:06.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
118322025-04-30T06:26:06.360ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c152c3b-9154-4f16-99e9-6bba9f755688", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118332025-04-30T06:26:06.360ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
118342025-04-30T06:26:06.362ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bedeee4b-c804-4336-a72b-474c2f57d53b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118352025-04-30T06:26:06.362ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
118362025-04-30T06:26:06.364ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "811e3a95-098c-429c-ba5d-d2e493c568ef", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118372025-04-30T06:26:06.364ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
118382025-04-30T06:26:06.368ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a189fd6d-04cd-4e71-853a-1796f4761418", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118392025-04-30T06:26:06.368ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
118402025-04-30T06:26:06.371ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "15aa983b-0fbb-4e08-9397-af2cb70a31c1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118412025-04-30T06:26:06.371ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
118422025-04-30T06:26:06.376ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bc0dbebb-3ce1-4239-8f6c-959ae7ce9b1e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118432025-04-30T06:26:06.377ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
118442025-04-30T06:26:06.378ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c5c92ee-527e-4326-9390-905e390634de", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118452025-04-30T06:26:06.378ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
118462025-04-30T06:26:06.380ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "156768a2-2584-498e-9370-986a72edb79d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118472025-04-30T06:26:06.380ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
118482025-04-30T06:26:06.381ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25151058-5c0c-4ad4-8fae-ede1bc1753e2", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118492025-04-30T06:26:06.381ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
118502025-04-30T06:26:06.382ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f11a9f2-988b-46d9-8c62-fea7a7929c73", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118512025-04-30T06:26:06.382ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
118522025-04-30T06:26:06.383ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "46f58246-17b7-45c0-8362-2ad7c630784d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118532025-04-30T06:26:06.383ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
118542025-04-30T06:26:06.384ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44336bc2-4373-41ea-bd40-8fdfedd373ce", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118552025-04-30T06:26:06.384ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
118562025-04-30T06:26:06.385ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "faea4c3b-33f9-480d-8c45-7f7e2645bb65", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118572025-04-30T06:26:06.386ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
118582025-04-30T06:26:06.387ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cb431bb-9348-46e3-949f-717304710c7b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118592025-04-30T06:26:06.387ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
118602025-04-30T06:26:06.389ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6db5fca1-ba8b-4659-b78e-0f42c5095c59", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118612025-04-30T06:26:06.389ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
118622025-04-30T06:26:06.395ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac46e6c0-8629-4df7-a07b-1359f3ec1493", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118632025-04-30T06:26:06.396ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
118642025-04-30T06:26:06.397ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118652025-04-30T06:26:06.398ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118662025-04-30T06:26:06.398ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118672025-04-30T06:26:06.398ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
118682025-04-30T06:26:06.400ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118692025-04-30T06:26:06.405ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118702025-04-30T06:26:06.422ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118712025-04-30T06:26:06.422ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118722025-04-30T06:26:06.422ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
118732025-04-30T06:26:06.424ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118742025-04-30T06:26:06.425ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118752025-04-30T06:26:06.425ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118762025-04-30T06:26:06.428ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118772025-04-30T06:26:06.428ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
118782025-04-30T06:26:06.429ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118792025-04-30T06:26:06.430ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118802025-04-30T06:26:06.431ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118812025-04-30T06:26:06.431ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118822025-04-30T06:26:06.431ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118832025-04-30T06:26:06.469ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e46ea536-cce7-4241-8720-60d5acaf680f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118842025-04-30T06:26:06.470ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
118852025-04-30T06:26:06.498ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
118862025-04-30T06:26:06.501ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62cc3d0e-db70-4f79-83f6-50884486de06", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118872025-04-30T06:26:06.501ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
118882025-04-30T06:26:06.503ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "810f627f-7f85-4f00-8575-834e7c3b22fe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118892025-04-30T06:26:06.503ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
118902025-04-30T06:26:06.504ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "edb6e55f-585d-4a4b-be3f-3b0674fbae9b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118912025-04-30T06:26:06.504ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
118922025-04-30T06:26:06.505ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec61155c-ac4b-4d49-867e-9c9d0157254e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118932025-04-30T06:26:06.505ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
118942025-04-30T06:26:06.506ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b48b8a66-b7a3-4406-913c-647e778226c9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118952025-04-30T06:26:06.507ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
118962025-04-30T06:26:06.508ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b18cda8-9321-4d64-8761-5614305f6da1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118972025-04-30T06:26:06.508ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
118982025-04-30T06:26:06.510ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c20b81-1490-4e68-b98f-46d4f36e03ad", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
118992025-04-30T06:26:06.510ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
119002025-04-30T06:26:06.514ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ea5f1d8-acfb-48ed-bb15-efc1d803ca32", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119012025-04-30T06:26:06.515ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
119022025-04-30T06:26:06.517ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bcca436-0134-4ca2-9e22-12b8422fc355", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119032025-04-30T06:26:06.517ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
119042025-04-30T06:26:06.519ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e56c22c7-08c3-4575-8413-c903516fa941", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119052025-04-30T06:26:06.519ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
119062025-04-30T06:26:06.523ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "763af081-cf01-466d-9eb4-6e3d8acbd856", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119072025-04-30T06:26:06.523ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
119082025-04-30T06:26:06.525ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c93cae3-6190-4139-9d54-13e372755687", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119092025-04-30T06:26:06.525ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
119102025-04-30T06:26:06.527ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0574c829-0916-4beb-936b-af9ae9dece5d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119112025-04-30T06:26:06.527ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
119122025-04-30T06:26:06.528ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83508a5b-2c57-4316-9594-ddc69fb04c51", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119132025-04-30T06:26:06.528ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
119142025-04-30T06:26:06.529ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c91fe18-be9b-4618-abbe-e04843cb093c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119152025-04-30T06:26:06.529ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
119162025-04-30T06:26:06.530ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "72b33871-080a-4a23-aaec-06f0d7a454af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119172025-04-30T06:26:06.530ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
119182025-04-30T06:26:06.531ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "57c57b24-7ae5-45bf-90a8-e9378e2940e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119192025-04-30T06:26:06.532ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
119202025-04-30T06:26:06.532ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
119212025-04-30T06:26:06.533ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6d76bf4-8577-4755-9a91-5e16f0454638", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119222025-04-30T06:26:06.534ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
119232025-04-30T06:26:06.535ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8533dae-7141-46f0-8a1e-b6abc6fca234", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119242025-04-30T06:26:06.535ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
119252025-04-30T06:26:06.536ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3cdf557-bdf8-4d09-96eb-bd48e478cd2e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119262025-04-30T06:26:06.536ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
119272025-04-30T06:26:06.537ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2f92111-c16f-4fb5-b924-d1029bc291a2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119282025-04-30T06:26:06.537ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
119292025-04-30T06:26:06.539ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b6bdf48-7871-47c3-a09f-66471f831fbf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119302025-04-30T06:26:06.539ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119312025-04-30T06:26:06.539ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119322025-04-30T06:26:06.539ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119332025-04-30T06:26:06.540ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119342025-04-30T06:26:06.541ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119352025-04-30T06:26:06.544ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119362025-04-30T06:26:06.547ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119372025-04-30T06:26:06.548ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119382025-04-30T06:26:06.548ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119392025-04-30T06:26:06.548ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119402025-04-30T06:26:06.548ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119412025-04-30T06:26:06.548ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119422025-04-30T06:26:06.549ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119432025-04-30T06:26:06.549ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119442025-04-30T06:26:06.549ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119452025-04-30T06:26:06.550ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
119462025-04-30T06:26:06.550ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119472025-04-30T06:26:06.550ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119482025-04-30T06:26:06.556ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119492025-04-30T06:26:06.556ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119502025-04-30T06:26:06.556ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119512025-04-30T06:26:06.557ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119522025-04-30T06:26:06.560ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119532025-04-30T06:26:06.569ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119542025-04-30T06:26:06.569ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119552025-04-30T06:26:06.569ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
119562025-04-30T06:26:06.570ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119572025-04-30T06:26:06.662ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
119582025-04-30T06:26:06.662ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
119592025-04-30T06:26:06.663ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119602025-04-30T06:26:06.663ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
119612025-04-30T06:26:06.664ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119622025-04-30T06:26:06.664ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:57563 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
119632025-04-30T06:26:06.665ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119642025-04-30T06:26:06.666ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119652025-04-30T06:26:06.667ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
119662025-04-30T06:26:06.670ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
119672025-04-30T06:26:06.670ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:57563 req_id = bedd98aa-29e0-4b89-9e11-378065f55f38 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
119682025-04-30T06:26:06.670ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1198 local_addr = [::1]:42668 method = GET remote_addr = [::1]:57563 req_id = bedd98aa-29e0-4b89-9e11-378065f55f38 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
119692025-04-30T06:26:06.671ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "bedd98aa-29e0-4b89-9e11-378065f55f38", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119702025-04-30T06:26:06.721ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
119712025-04-30T06:26:06.721ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
119722025-04-30T06:26:06.721ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:58042 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
119732025-04-30T06:26:06.722ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
119742025-04-30T06:26:06.722ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
119752025-04-30T06:26:06.722ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:58042 req_id = d546fd21-076f-44c9-94f6-e28f182b0bc0 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
119762025-04-30T06:26:06.722ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 221 local_addr = [::1]:47918 method = GET remote_addr = [::1]:58042 req_id = d546fd21-076f-44c9-94f6-e28f182b0bc0 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
119772025-04-30T06:26:06.723ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d546fd21-076f-44c9-94f6-e28f182b0bc0", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119782025-04-30T06:26:06.786ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
119792025-04-30T06:26:06.786ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
119802025-04-30T06:26:06.786ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:60513 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
119812025-04-30T06:26:06.790ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
119822025-04-30T06:26:06.790ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:60513 req_id = f6a6067b-4767-421a-8405-255eee4c242d sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
119832025-04-30T06:26:06.793ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
119842025-04-30T06:26:06.793ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119852025-04-30T06:26:06.793ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 1975 local_addr = [::1]:35744 method = GET remote_addr = [::1]:60513 req_id = f6a6067b-4767-421a-8405-255eee4c242d response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
119862025-04-30T06:26:06.793ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a6067b-4767-421a-8405-255eee4c242d", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119872025-04-30T06:26:06.856ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
119882025-04-30T06:26:06.856ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
119892025-04-30T06:26:06.857ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:51612 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119902025-04-30T06:26:06.858ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:51612 req_id = 38f44d90-ec1d-4d4f-be59-256e1be911a4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119912025-04-30T06:26:06.858ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 587 local_addr = [::1]:35343 method = GET remote_addr = [::1]:51612 req_id = 38f44d90-ec1d-4d4f-be59-256e1be911a4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
119922025-04-30T06:26:06.859ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "38f44d90-ec1d-4d4f-be59-256e1be911a4", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:05 GMT"} })
119932025-04-30T06:26:06.862ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119942025-04-30T06:26:06.862ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
119952025-04-30T06:26:06.862ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
119962025-04-30T06:26:06.872ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
119972025-04-30T06:26:06.873ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119982025-04-30T06:26:06.873ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119992025-04-30T06:26:06.873ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120002025-04-30T06:26:06.874ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120012025-04-30T06:26:06.879ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120022025-04-30T06:26:06.888ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120032025-04-30T06:26:06.888ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120042025-04-30T06:26:06.888ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120052025-04-30T06:26:06.891ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120062025-04-30T06:26:06.908ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
120072025-04-30T06:26:06.944ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120082025-04-30T06:26:06.944ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120092025-04-30T06:26:06.944ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 833.063825ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
120102025-04-30T06:26:06.944ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
120112025-04-30T06:26:06.944ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120122025-04-30T06:26:06.944ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120132025-04-30T06:26:06.944ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120142025-04-30T06:26:06.944ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120152025-04-30T06:26:06.945ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120162025-04-30T06:26:06.948ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120172025-04-30T06:26:06.953ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120182025-04-30T06:26:06.953ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120192025-04-30T06:26:06.953ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120202025-04-30T06:26:06.954ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120212025-04-30T06:26:06.968ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
120222025-04-30T06:26:06.968ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120232025-04-30T06:26:06.968ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120242025-04-30T06:26:06.968ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
120252025-04-30T06:26:06.969ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120262025-04-30T06:26:06.969ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
120272025-04-30T06:26:06.969ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120282025-04-30T06:26:06.970ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e8 d3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51151 req_id = adbc96fe-0d65-423f-9141-8727dadad81c
120292025-04-30T06:26:06.970ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59603,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51151 req_id = adbc96fe-0d65-423f-9141-8727dadad81c
120302025-04-30T06:26:06.970ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
120312025-04-30T06:26:06.970ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
120322025-04-30T06:26:06.971ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120332025-04-30T06:26:06.971ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51151 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = adbc96fe-0d65-423f-9141-8727dadad81c
120342025-04-30T06:26:06.971ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3231, tv_nsec: 770580619 } })
120352025-04-30T06:26:06.971ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
120362025-04-30T06:26:06.972ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
120372025-04-30T06:26:06.972ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 71 44 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e qD..........-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63609 req_id = 4678db20-79c9-4a44-9889-9a380138ae83
120382025-04-30T06:26:06.972ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28996,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63609 req_id = 4678db20-79c9-4a44-9889-9a380138ae83
120392025-04-30T06:26:06.972ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
120402025-04-30T06:26:06.973ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63609 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4678db20-79c9-4a44-9889-9a380138ae83
120412025-04-30T06:26:06.973ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 18 0c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63067 req_id = d7bfd2bf-84e7-4e5f-93a7-83d905e283f8
120422025-04-30T06:26:06.973ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6156,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63067 req_id = d7bfd2bf-84e7-4e5f-93a7-83d905e283f8
120432025-04-30T06:26:06.973ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120442025-04-30T06:26:06.973ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63067 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d7bfd2bf-84e7-4e5f-93a7-83d905e283f8
120452025-04-30T06:26:07.136ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
120462025-04-30T06:26:07.137ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120472025-04-30T06:26:07.149ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
120482025-04-30T06:26:07.149ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120492025-04-30T06:26:07.149ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
120502025-04-30T06:26:07.150ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120512025-04-30T06:26:07.151ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: db 76 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .v..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59272 req_id = 24b5ca5e-9a8a-4513-81ef-008903d1e8ee
120522025-04-30T06:26:07.152ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56182,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59272 req_id = 24b5ca5e-9a8a-4513-81ef-008903d1e8ee
120532025-04-30T06:26:07.153ZDEBGtest_replacement_sanity (store): query key key = _clickhouse-admin-keeper._tcp
120542025-04-30T06:26:07.153ZERROtest_replacement_sanity (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56182, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59272 req_id = 24b5ca5e-9a8a-4513-81ef-008903d1e8ee
120552025-04-30T06:26:07.155ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection background_task = inventory_collection
120562025-04-30T06:26:07.155ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:40918
120572025-04-30T06:26:07.155ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/ignition
120582025-04-30T06:26:07.155ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120592025-04-30T06:26:07.155ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
120602025-04-30T06:26:07.156ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120612025-04-30T06:26:07.167ZDEBGtest_replacement_sanity (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
120622025-04-30T06:26:07.168ZWARNtest_replacement_sanity (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
120632025-04-30T06:26:07.175ZWARNtest_replacement_sanity (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
120642025-04-30T06:26:07.180ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b14bfd2-7c94-4f18-82f7-7c97e16e8cd2", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120652025-04-30T06:26:07.181ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0
120662025-04-30T06:26:07.191ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8604879-0c23-4cb5-ae37-4077121e9465", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120672025-04-30T06:26:07.192ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0
120682025-04-30T06:26:07.204ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b5d9367-9d39-434b-a0c4-704bf2f339dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120692025-04-30T06:26:07.204ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1
120702025-04-30T06:26:07.206ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af178046-5d98-405d-986b-54190f8a0e4f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120712025-04-30T06:26:07.206ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0
120722025-04-30T06:26:07.212ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a2577c2-7fdc-4ed8-b2ce-6595d8851f9a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120732025-04-30T06:26:07.212ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1
120742025-04-30T06:26:07.216ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f600273f-792e-404b-bbe2-36fcce68fda0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120752025-04-30T06:26:07.216ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0
120762025-04-30T06:26:07.219ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
120772025-04-30T06:26:07.241ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90c80802-06dd-4651-ac60-14925f822597", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120782025-04-30T06:26:07.242ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1
120792025-04-30T06:26:07.249ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ee8b855-6858-4747-8e9b-fdf966139ca2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120802025-04-30T06:26:07.249ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cmpa
120812025-04-30T06:26:07.249ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3627ae9b-439a-4491-92d4-8f57a58c9dc4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120822025-04-30T06:26:07.249ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
120832025-04-30T06:26:07.249ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d3a388-6587-4c6d-bdc9-e8c767b0fea8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120842025-04-30T06:26:07.249ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
120852025-04-30T06:26:07.250ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cbfd574-f839-41f9-9a0a-f41813492ed7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120862025-04-30T06:26:07.250ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/0/component/rot/cfpa
120872025-04-30T06:26:07.252ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19fa7077-75a0-4349-b011-a6505005b6c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120882025-04-30T06:26:07.252ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1
120892025-04-30T06:26:07.254ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3ab506e-5e78-49f5-981e-5d39e769ba21", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120902025-04-30T06:26:07.254ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0
120912025-04-30T06:26:07.257ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6ff9cc2-61fc-4a52-84e5-8ad1df268b07", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120922025-04-30T06:26:07.257ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1
120932025-04-30T06:26:07.258ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b366b724-9081-49d3-a9c1-b3d8b8e77fbf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120942025-04-30T06:26:07.259ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0
120952025-04-30T06:26:07.261ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa4ad529-d939-4e45-b12c-009be2cd6a75", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120962025-04-30T06:26:07.261ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1
120972025-04-30T06:26:07.263ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8838efe-a97a-4903-92b0-8edceaff045d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
120982025-04-30T06:26:07.263ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0
120992025-04-30T06:26:07.264ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6712621-acb7-47d7-819e-32158e089bf0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121002025-04-30T06:26:07.265ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1
121012025-04-30T06:26:07.266ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a549676c-8e1a-4604-96f4-d413318e4396", "content-length": "179", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121022025-04-30T06:26:07.266ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cmpa
121032025-04-30T06:26:07.267ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89fc32f5-d97d-42e8-b514-ddab4a84092a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121042025-04-30T06:26:07.267ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
121052025-04-30T06:26:07.273ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121062025-04-30T06:26:07.273ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121072025-04-30T06:26:07.273ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121082025-04-30T06:26:07.273ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
121092025-04-30T06:26:07.274ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121102025-04-30T06:26:07.284ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f68298c-c17d-41c7-9461-1dfa65b4435c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121112025-04-30T06:26:07.285ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
121122025-04-30T06:26:07.285ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121132025-04-30T06:26:07.304ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121142025-04-30T06:26:07.304ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121152025-04-30T06:26:07.304ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
121162025-04-30T06:26:07.305ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121172025-04-30T06:26:07.306ZINFOtest_replacement_sanity (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
121182025-04-30T06:26:07.307ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
121192025-04-30T06:26:07.307ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6f21f77f-6eba-41b1-82ce-76830eb546e3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
121202025-04-30T06:26:07.308ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6f21f77f-6eba-41b1-82ce-76830eb546e3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
121212025-04-30T06:26:07.308ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6f21f77f-6eba-41b1-82ce-76830eb546e3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
121222025-04-30T06:26:07.308ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6f21f77f-6eba-41b1-82ce-76830eb546e3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
121232025-04-30T06:26:07.309ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a649010-d5c8-42c8-a43e-ad1e1d338c69", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121242025-04-30T06:26:07.309ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/switch/1/component/rot/cfpa
121252025-04-30T06:26:07.310ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121262025-04-30T06:26:07.310ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
121272025-04-30T06:26:07.311ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121282025-04-30T06:26:07.311ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b844d934-6634-4d92-8703-08143379b884", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121292025-04-30T06:26:07.322ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0
121302025-04-30T06:26:07.322ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 14674 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = 6f21f77f-6eba-41b1-82ce-76830eb546e3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
121312025-04-30T06:26:07.323ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f21f77f-6eba-41b1-82ce-76830eb546e3", "content-length": "405", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121322025-04-30T06:26:07.323ZDEBGtest_replacement_sanity (oximeter): client request body = None method = GET uri = http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
121332025-04-30T06:26:07.326ZTRACtest_replacement_sanity (dropshot_internal): incoming request local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = f3367f26-0881-4c98-bad4-cab674c10f1d uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
121342025-04-30T06:26:07.326ZTRACtest_replacement_sanity (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = f3367f26-0881-4c98-bad4-cab674c10f1d resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
121352025-04-30T06:26:07.326ZDEBGtest_replacement_sanity (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = f3367f26-0881-4c98-bad4-cab674c10f1d roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
121362025-04-30T06:26:07.326ZDEBGtest_replacement_sanity (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = f3367f26-0881-4c98-bad4-cab674c10f1d resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
121372025-04-30T06:26:07.327ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "444a1a6a-9e71-4ada-bb57-5bc92457061e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121382025-04-30T06:26:07.328ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0
121392025-04-30T06:26:07.345ZINFOtest_replacement_sanity (dropshot_internal): request completed latency_us = 20048 local_addr = 127.0.0.1:43527 method = GET remote_addr = 127.0.0.1:64764 req_id = f3367f26-0881-4c98-bad4-cab674c10f1d response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D
121402025-04-30T06:26:07.346ZDEBGtest_replacement_sanity (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:43527/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJjMzFkYTBhZS05MzBkLTRkZDktYTg5Ny0xNTlkNWE2Y2Q0MjAifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3367f26-0881-4c98-bad4-cab674c10f1d", "content-length": "29", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121412025-04-30T06:26:07.346ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121422025-04-30T06:26:07.346ZTRACtest_replacement_sanity (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = c31da0ae-930d-4dd9-a897-159d5a6cd420
121432025-04-30T06:26:07.346ZINFOtest_replacement_sanity (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
121442025-04-30T06:26:07.355ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51e6b660-fa01-46dc-9095-cdd05279dc45", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121452025-04-30T06:26:07.355ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1
121462025-04-30T06:26:07.358ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cfcd5aa3-5568-44c4-96d4-c01b317c695e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121472025-04-30T06:26:07.359ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0
121482025-04-30T06:26:07.362ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3abdabd5-6d48-4f5a-ae51-a1d9c3c453aa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121492025-04-30T06:26:07.363ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1
121502025-04-30T06:26:07.375ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f44c96fc-9ace-4ac2-9106-ab968c6ab73e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121512025-04-30T06:26:07.383ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0
121522025-04-30T06:26:07.387ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "869bed8d-2c9d-4ec9-9bea-496eeaf906c1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121532025-04-30T06:26:07.387ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1
121542025-04-30T06:26:07.394ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c032e25a-f786-4cce-b36e-058d2f72c126", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121552025-04-30T06:26:07.394ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cmpa
121562025-04-30T06:26:07.395ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e06ff5c0-e985-418b-9701-e39b912cf30b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121572025-04-30T06:26:07.396ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
121582025-04-30T06:26:07.398ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
121592025-04-30T06:26:07.398ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "73990b03-9948-48ed-aeab-c09d072379d8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121602025-04-30T06:26:07.398ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
121612025-04-30T06:26:07.400ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17a94854-6e19-406f-ad04-d309deeb3a4d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121622025-04-30T06:26:07.400ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/0/component/rot/cfpa
121632025-04-30T06:26:07.402ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19169dcd-d877-454c-94c8-a52b6aca0b93", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121642025-04-30T06:26:07.402ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1
121652025-04-30T06:26:07.404ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eac13100-3fa0-4492-a24e-f40033c1d56c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121662025-04-30T06:26:07.404ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0
121672025-04-30T06:26:07.408ZDEBGtest_replacement_sanity (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121682025-04-30T06:26:07.408ZDEBGtest_replacement_sanity (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121692025-04-30T06:26:07.410ZTRACtest_replacement_sanity (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121702025-04-30T06:26:07.411ZDEBGtest_replacement_sanity (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121712025-04-30T06:26:07.411ZTRACtest_replacement_sanity (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121722025-04-30T06:26:07.411ZTRACtest_replacement_sanity (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
121732025-04-30T06:26:07.413ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fdec0e0-28b2-417d-9a55-03abc18581d2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121742025-04-30T06:26:07.413ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1
121752025-04-30T06:26:07.417ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3a2d10d-f87d-40be-a2d1-44d6234cd39e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121762025-04-30T06:26:07.417ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0
121772025-04-30T06:26:07.422ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "691ae315-d8a1-4ca5-a6c9-bf290fc37a0b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121782025-04-30T06:26:07.422ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1
121792025-04-30T06:26:07.425ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbf7038f-362f-42f9-baa6-f100171e9526", "content-length": "169", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121802025-04-30T06:26:07.425ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0
121812025-04-30T06:26:07.427ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "793931f8-c335-4cf0-ae6c-5ed74e52fc91", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121822025-04-30T06:26:07.428ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1
121832025-04-30T06:26:07.449ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121842025-04-30T06:26:07.450ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121852025-04-30T06:26:07.450ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121862025-04-30T06:26:07.450ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
121872025-04-30T06:26:07.451ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121882025-04-30T06:26:07.455ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121892025-04-30T06:26:07.460ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121902025-04-30T06:26:07.460ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121912025-04-30T06:26:07.460ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
121922025-04-30T06:26:07.461ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121932025-04-30T06:26:07.472ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121942025-04-30T06:26:07.472ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
121952025-04-30T06:26:07.472ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121962025-04-30T06:26:07.498ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa0bd0bc-f595-4257-8a32-cf51189939a5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121972025-04-30T06:26:07.499ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cmpa
121982025-04-30T06:26:07.504ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6778348d-7b22-4cc1-afad-c537ee93c6c1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
121992025-04-30T06:26:07.504ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
122002025-04-30T06:26:07.507ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1513eb9-87e9-4b04-8f37-f960e5467a3c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122012025-04-30T06:26:07.508ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
122022025-04-30T06:26:07.509ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "850483a8-26b1-4719-89a2-5853d581951f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122032025-04-30T06:26:07.510ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:40918 method = GET uri = http://[::1]:40918/sp/sled/1/component/rot/cfpa
122042025-04-30T06:26:07.511ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:40918 result = Ok(Response { url: "http://[::1]:40918/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f18fecd9-10bf-48ca-ae3b-131d2d1691d6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122052025-04-30T06:26:07.511ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:50113
122062025-04-30T06:26:07.512ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/ignition
122072025-04-30T06:26:07.513ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8d9a24b-4901-48d9-9acd-70b49f25782f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122082025-04-30T06:26:07.513ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/0
122092025-04-30T06:26:07.515ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "688f711b-0a6c-4e09-a7b0-a649463bc077", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122102025-04-30T06:26:07.515ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/switch/1
122112025-04-30T06:26:07.517ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "852c9939-a1df-48a2-a405-55615e90ba2c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122122025-04-30T06:26:07.518ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/0
122132025-04-30T06:26:07.524ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0403447a-1f54-4fb6-93b7-82e1dda962f7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122142025-04-30T06:26:07.524ZDEBGtest_replacement_sanity (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:50113 method = GET uri = http://[::1]:50113/sp/sled/1
122152025-04-30T06:26:07.526ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
122162025-04-30T06:26:07.527ZDEBGtest_replacement_sanity (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:50113 result = Ok(Response { url: "http://[::1]:50113/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f81ee2ed-e83b-4589-b4a6-cd5390dbf92e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122172025-04-30T06:26:07.527ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122182025-04-30T06:26:07.527ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122192025-04-30T06:26:07.527ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122202025-04-30T06:26:07.527ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122212025-04-30T06:26:07.535ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122222025-04-30T06:26:07.540ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122232025-04-30T06:26:07.556ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122242025-04-30T06:26:07.556ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122252025-04-30T06:26:07.556ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122262025-04-30T06:26:07.556ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122272025-04-30T06:26:07.556ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122282025-04-30T06:26:07.557ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122292025-04-30T06:26:07.564ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122302025-04-30T06:26:07.572ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122312025-04-30T06:26:07.572ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122322025-04-30T06:26:07.572ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122332025-04-30T06:26:07.573ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122342025-04-30T06:26:07.577ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122352025-04-30T06:26:07.577ZTRACtest_replacement_sanity (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122362025-04-30T06:26:07.577ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122372025-04-30T06:26:07.577ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
122382025-04-30T06:26:07.578ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122392025-04-30T06:26:07.647ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:42668
122402025-04-30T06:26:07.647ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:42668 background_task = inventory_collection body = None method = GET uri = http://[::1]:42668/inventory
122412025-04-30T06:26:07.648ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:42668 remote_addr = [::1]:57550 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913
122422025-04-30T06:26:07.648ZDEBGtest_replacement_sanity (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122432025-04-30T06:26:07.648ZDEBGtest_replacement_sanity (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122442025-04-30T06:26:07.648ZDEBGtest_replacement_sanity (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833
122452025-04-30T06:26:07.649ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
122462025-04-30T06:26:07.650ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122472025-04-30T06:26:07.655ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122482025-04-30T06:26:07.656ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122492025-04-30T06:26:07.656ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
122502025-04-30T06:26:07.656ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122512025-04-30T06:26:07.656ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:42668 method = GET remote_addr = [::1]:57550 req_id = ff9b67e9-d7ab-4d73-b17f-cf0a3a871ef2 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
122522025-04-30T06:26:07.657ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 492 local_addr = [::1]:42668 method = GET remote_addr = [::1]:57550 req_id = ff9b67e9-d7ab-4d73-b17f-cf0a3a871ef2 response_code = 200 sled_id = 039be560-54cc-49e3-88df-1a29dadbf913 uri = /inventory
122532025-04-30T06:26:07.657ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:42668 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:42668/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff9b67e9-d7ab-4d73-b17f-cf0a3a871ef2", "content-length": "811", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122542025-04-30T06:26:07.706ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:47918
122552025-04-30T06:26:07.706ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:47918 background_task = inventory_collection body = None method = GET uri = http://[::1]:47918/inventory
122562025-04-30T06:26:07.706ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:47918 remote_addr = [::1]:54631 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d
122572025-04-30T06:26:07.707ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 2 table_name = fields_string
122582025-04-30T06:26:07.707ZTRACtest_replacement_sanity (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
122592025-04-30T06:26:07.707ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:47918 method = GET remote_addr = [::1]:54631 req_id = 5624fa5b-0731-44fa-896a-0a4f1636f9c6 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
122602025-04-30T06:26:07.708ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 412 local_addr = [::1]:47918 method = GET remote_addr = [::1]:54631 req_id = 5624fa5b-0731-44fa-896a-0a4f1636f9c6 response_code = 200 sled_id = 2823225f-fa9c-49ed-b0f5-744d993ad37d uri = /inventory
122612025-04-30T06:26:07.708ZTRACtest_replacement_sanity (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122622025-04-30T06:26:07.708ZDEBGtest_replacement_sanity (InternalApi): roles roles = RoleSet { roles: {} }
122632025-04-30T06:26:07.709ZDEBGtest_replacement_sanity (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122642025-04-30T06:26:07.709ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:47918 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:47918/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5624fa5b-0731-44fa-896a-0a4f1636f9c6", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122652025-04-30T06:26:07.771ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35744
122662025-04-30T06:26:07.771ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35744 background_task = inventory_collection body = None method = GET uri = http://[::1]:35744/inventory
122672025-04-30T06:26:07.772ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35744 remote_addr = [::1]:44126 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da
122682025-04-30T06:26:07.777ZWARNtest_replacement_sanity (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
122692025-04-30T06:26:07.777ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35744 method = GET remote_addr = [::1]:44126 req_id = 82485cf5-237a-438b-bb27-e2d6f3aa0e3c sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
122702025-04-30T06:26:07.778ZDEBGtest_replacement_sanity (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = b8acd62a-f986-4471-9673-45282afe5833 n_rows = 1 table_name = measurements_i64
122712025-04-30T06:26:07.778ZTRACtest_replacement_sanity (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122722025-04-30T06:26:07.778ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 724 local_addr = [::1]:35744 method = GET remote_addr = [::1]:44126 req_id = 82485cf5-237a-438b-bb27-e2d6f3aa0e3c response_code = 200 sled_id = 98fd4d7a-8f38-420e-b1c6-ed6e4b0bb7da uri = /inventory
122732025-04-30T06:26:07.778ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35744 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35744/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "82485cf5-237a-438b-bb27-e2d6f3aa0e3c", "content-length": "439", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122742025-04-30T06:26:07.845ZDEBGtest_replacement_sanity (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:35343
122752025-04-30T06:26:07.845ZDEBGtest_replacement_sanity (BackgroundTasks): client request SledAgent = http://[::1]:35343 background_task = inventory_collection body = None method = GET uri = http://[::1]:35343/inventory
122762025-04-30T06:26:07.846ZINFOtest_replacement_sanity (dropshot): accepted connection local_addr = [::1]:35343 remote_addr = [::1]:61439 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122772025-04-30T06:26:07.847ZTRACtest_replacement_sanity (dropshot): incoming request local_addr = [::1]:35343 method = GET remote_addr = [::1]:61439 req_id = fa601caf-aff3-4282-a184-466238d58bf9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122782025-04-30T06:26:07.848ZINFOtest_replacement_sanity (dropshot): request completed latency_us = 893 local_addr = [::1]:35343 method = GET remote_addr = [::1]:61439 req_id = fa601caf-aff3-4282-a184-466238d58bf9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122792025-04-30T06:26:07.848ZDEBGtest_replacement_sanity (BackgroundTasks): client response SledAgent = http://[::1]:35343 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:35343/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa601caf-aff3-4282-a184-466238d58bf9", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:26:07 GMT"} })
122802025-04-30T06:26:07.854ZDEBGtest_replacement_sanity (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122812025-04-30T06:26:07.854ZDEBGtest_replacement_sanity (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
122822025-04-30T06:26:07.854ZDEBGtest_replacement_sanity (BackgroundTasks): finished collection background_task = inventory_collection
122832025-04-30T06:26:07.859ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122842025-04-30T06:26:07.859ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122852025-04-30T06:26:07.859ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122862025-04-30T06:26:07.859ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122872025-04-30T06:26:07.860ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122882025-04-30T06:26:07.864ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122892025-04-30T06:26:07.868ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
122902025-04-30T06:26:07.868ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122912025-04-30T06:26:07.868ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122922025-04-30T06:26:07.869ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122932025-04-30T06:26:07.901ZDEBGtest_replacement_sanity: attempt to fetch latest inventory collection result = Ok("found")
122942025-04-30T06:26:07.912ZWARNtest_replacement_sanity (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
122952025-04-30T06:26:07.912ZWARNtest_replacement_sanity (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
122962025-04-30T06:26:07.912ZDEBGtest_replacement_sanity (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 968.26284ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
122972025-04-30T06:26:07.912ZDEBGtest_replacement_sanity (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
122982025-04-30T06:26:07.912ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
122992025-04-30T06:26:07.913ZTRACtest_replacement_sanity (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123002025-04-30T06:26:07.913ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123012025-04-30T06:26:07.913ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123022025-04-30T06:26:07.914ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123032025-04-30T06:26:07.919ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123042025-04-30T06:26:07.922ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123052025-04-30T06:26:07.923ZTRACtest_replacement_sanity (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123062025-04-30T06:26:07.923ZDEBGtest_replacement_sanity (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123072025-04-30T06:26:07.923ZDEBGtest_replacement_sanity (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123082025-04-30T06:26:07.926ZDEBGtest_replacement_sanity (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(8e0b9a02-fe6f-4a9f-9818-5453e787b6db, 0)]
123092025-04-30T06:26:07.926ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
123102025-04-30T06:26:07.927ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 3a d9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 :............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65309 req_id = 52232cff-dc6e-459e-85bf-35861fc6bc36
123112025-04-30T06:26:07.927ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15065,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65309 req_id = 52232cff-dc6e-459e-85bf-35861fc6bc36
123122025-04-30T06:26:07.928ZDEBGtest_replacement_sanity (store): query key key = _mgs._tcp
123132025-04-30T06:26:07.928ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
123142025-04-30T06:26:07.928ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123152025-04-30T06:26:07.928ZDEBGtest_replacement_sanity (dns): dns response additional_records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65309 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 52232cff-dc6e-459e-85bf-35861fc6bc36
123162025-04-30T06:26:07.929ZDEBGtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 40918, target: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 50113, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3232, tv_nsec: 727889134 } })
123172025-04-30T06:26:07.929ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
123182025-04-30T06:26:07.929ZTRACtest_replacement_sanity (DnsResolver): lookup_service_targets: looking up SRV target
123192025-04-30T06:26:07.929ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d8 87 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 30 33 39 62 65 35 36 30 2d 35 drite-039be560-5\n0020: 34 63 63 2d 34 39 65 33 2d 38 38 64 66 2d 31 61 4cc-49e3-88df-1a\n0030: 32 39 64 61 64 62 66 39 31 33 04 68 6f 73 74 0d 29dadbf913.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59534 req_id = 4624738c-9488-48bc-a8a5-a981da9913c8
123202025-04-30T06:26:07.929ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55431,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 48,\n 51,\n 57,\n 98,\n 101,\n 53,\n 54,\n 48,\n 45,\n 53,\n 52,\n 99,\n 99,\n 45,\n 52,\n 57,\n 101,\n 51,\n 45,\n 56,\n 56,\n 100,\n 102,\n 45,\n 49,\n 97,\n 50,\n 57,\n 100,\n 97,\n 100,\n 98,\n 102,\n 57,\n 49,\n 51,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59534 req_id = 4624738c-9488-48bc-a8a5-a981da9913c8
123212025-04-30T06:26:07.929ZDEBGtest_replacement_sanity (store): query key key = dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host
123222025-04-30T06:26:07.929ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59534 query = LowerQuery { name: LowerName(Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-039be560-54cc-49e3-88df-1a29dadbf913.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4624738c-9488-48bc-a8a5-a981da9913c8
123232025-04-30T06:26:07.929ZTRACtest_replacement_sanity (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d8 6e 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .n..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57612 req_id = d6e51f0b-7be2-4056-af3c-2bebe6cab4a1
123242025-04-30T06:26:07.930ZDEBGtest_replacement_sanity (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55406,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57612 req_id = d6e51f0b-7be2-4056-af3c-2bebe6cab4a1
123252025-04-30T06:26:07.930ZDEBGtest_replacement_sanity (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123262025-04-30T06:26:07.930ZDEBGtest_replacement_sanity (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57612 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d6e51f0b-7be2-4056-af3c-2bebe6cab4a1
123272025-04-30T06:26:08.120ZTRACtest_replacement_sanity (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal