Max level shown:
12025-04-30T06:35:21.883ZDEBGtest_metrics (init_with_steps): Running step start_crdb
22025-04-30T06:35:21.884ZDEBGtest_metrics: Starting CRDB
32025-04-30T06:35:21.917ZINFOtest_metrics: cockroach temporary directory: /var/tmp/omicron_tmp/.tmp9NGJpe
42025-04-30T06:35:21.917ZINFOtest_metrics: cockroach: copying from seed tarball (/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar) to storage directory (/var/tmp/omicron_tmp/.tmp9NGJpe/data)
52025-04-30T06:35:21.936ZINFOtest_metrics: cockroach command line: cockroach start-single-node --insecure --http-addr=:0 --store=path=/var/tmp/omicron_tmp/.tmp9NGJpe/data,ballast-size=0 --listen-addr [::1]:0 --listening-url-file /var/tmp/omicron_tmp/.tmp9NGJpe/listen-url
62025-04-30T06:35:21.936ZINFOtest_metrics: cockroach environment: BUILDOMAT_JOB_ID=01JT2JWKGHF3JHFMW0JR13NSA6 BUILDOMAT_TASK_ID=4 CARGO=/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/bin/cargo CARGO_HOME=/home/build/.cargo CARGO_INCREMENTAL=0 CARGO_MANIFEST_DIR=/work/oxidecomputer/omicron/nexus CARGO_PKG_AUTHORS= CARGO_PKG_DESCRIPTION= CARGO_PKG_HOMEPAGE= CARGO_PKG_LICENSE=MPL-2.0 CARGO_PKG_LICENSE_FILE= CARGO_PKG_NAME=omicron-nexus CARGO_PKG_REPOSITORY= CARGO_PKG_RUST_VERSION= CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE= CARGO_TERM_COLOR=always CI=true CRDB_SEED_TAR=/var/tmp/omicron_tmp/crdb-base-build/f234518892194a05ad8767dc97f01a61a55baf1431942a314ce8b0f72b18ec2e.tar GITHUB_REPOSITORY=oxidecomputer/omicron GITHUB_SHA=531912b3fc900900bc7bd18f5eac1ce1e22cb1cd GOTRACEBACK=crash HOME=/home/build LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib LOGNAME=build NEXTEST=1 NEXTEST_BIN_EXE_nexus=/work/oxidecomputer/omicron/target/debug/nexus NEXTEST_BIN_EXE_schema-updater=/work/oxidecomputer/omicron/target/debug/schema-updater NEXTEST_EXECUTION_MODE=process-per-test NEXTEST_LD_LIBRARY_PATH=/work/oxidecomputer/omicron/target/debug/build/aws-lc-sys-6532f52db341a7cc/out/build/artifacts:/work/oxidecomputer/omicron/target/debug/build/blake3-403ee89fc24db8f8/out:/work/oxidecomputer/omicron/target/debug/build/bzip2-sys-aece66bfb5c8df92/out/lib:/work/oxidecomputer/omicron/target/debug/build/libgit2-sys-5fe90ecb72e1690f/out/build:/work/oxidecomputer/omicron/target/debug/build/ring-fb368fe91b2bd145/out:/work/oxidecomputer/omicron/target/debug/build/tofino-ffe28f7f2301d0c2/out:/work/oxidecomputer/omicron/target/debug/build/zstd-sys-a7fb19a1e17b70f6/out:/work/oxidecomputer/omicron/target/debug/deps:/work/oxidecomputer/omicron/target/debug:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib/rustlib/x86_64-unknown-illumos/lib:/home/build/.rustup/toolchains/1.86.0-x86_64-unknown-illumos/lib NEXTEST_PROFILE=ci NEXTEST_RUN_ID=29735cc2-0883-4059-8ca2-1da53f01ede3 NEXTEST_TEST_GLOBAL_SLOT=2 NEXTEST_TEST_GROUP=@global NEXTEST_TEST_GROUP_SLOT=none OUT_DIR=/work/oxidecomputer/omicron/target/debug/build/omicron-nexus-553062f859dbbfa0/out PATH=/work/oxidecomputer/omicron/out/mgd/root/opt/oxide/mgd/bin:/work/oxidecomputer/omicron/out/dendrite-stub/bin:/work/oxidecomputer/omicron/out/clickhouse:/work/oxidecomputer/omicron/out/cockroachdb/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin PWD=/work/oxidecomputer/omicron RUSTC_BOOTSTRAP=1 RUSTDOCFLAGS=--document-private-items -D warnings RUSTFLAGS=-D warnings -C link-arg=-R/usr/platform/oxide/lib/amd64 RUSTUP_HOME=/home/build/.rustup RUSTUP_TOOLCHAIN=1.86.0-x86_64-unknown-illumos RUST_BACKTRACE=1 RUST_RECURSION_COUNT=1 SHLVL=1 SSL_CERT_DIR=/usr/ssl/certs SSL_CERT_FILE=/etc/ssl/cacert.pem TMPDIR=/var/tmp/omicron_tmp TZ=UTC USER=build _=/usr/bin/ptime __NEXTEST_ATTEMPT=1
72025-04-30T06:35:23.516ZINFOtest_metrics: cockroach pid: 24731
82025-04-30T06:35:23.516ZINFOtest_metrics: cockroach listen URL: postgresql://root@[::1]:57004/omicron?sslmode=disable
92025-04-30T06:35:23.517ZDEBGtest_metrics (init_with_steps): Running step start_clickhouse
102025-04-30T06:35:23.517ZDEBGtest_metrics: Starting Clickhouse
112025-04-30T06:35:24.269ZDEBGtest_metrics (init_with_steps): Running step start_gateway_switch0
122025-04-30T06:35:24.269ZDEBGtest_metrics: Starting Management Gateway
132025-04-30T06:35:24.395ZDEBGtest_metrics (init_with_steps): Running step start_dendrite_switch0
142025-04-30T06:35:24.395ZDEBGtest_metrics: Starting Dendrite for switch0
152025-04-30T06:35:24.455ZDEBGtest_metrics (init_with_steps): Running step start_mgd_switch0
162025-04-30T06:35:24.455ZDEBGtest_metrics: Starting mgd for switch0
172025-04-30T06:35:24.574ZDEBGtest_metrics: mgd port is 39463
182025-04-30T06:35:24.575ZDEBGtest_metrics (init_with_steps): Running step record_switch_dns
192025-04-30T06:35:24.575ZDEBGtest_metrics: Recording DNS for the switch zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 switch_location = switch0
202025-04-30T06:35:24.575ZDEBGtest_metrics (init_with_steps): Running step start_internal_dns
212025-04-30T06:35:24.575ZINFOtest_metrics (store): opening sled database path = /var/tmp/omicron_tmp/.tmpsdpBnr
222025-04-30T06:35:24.683ZINFOtest_metrics (store): pruning trees for generations newer than 0
232025-04-30T06:35:24.684ZINFOtest_metrics (store): pruning trees for generations older than 0 keep = 3
242025-04-30T06:35:24.684ZINFOtest_metrics (dns): DNS server bound to address kind = dns local_address = [::1]:53219
252025-04-30T06:35:24.685ZDEBGtest_metrics (http): registered endpoint kind = dns local_addr = [::1]:36868 method = GET path = /config versions = all
262025-04-30T06:35:24.685ZDEBGtest_metrics (http): registered endpoint kind = dns local_addr = [::1]:36868 method = PUT path = /config versions = all
272025-04-30T06:35:24.685ZINFOtest_metrics (http): listening kind = dns local_addr = [::1]:36868
282025-04-30T06:35:24.686ZDEBGtest_metrics (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:36868
292025-04-30T06:35:24.687ZDEBGtest_metrics (init_with_steps): Running step start_external_dns
302025-04-30T06:35:24.687ZINFOtest_metrics (store): opening sled database path = /var/tmp/omicron_tmp/.tmpl1wqp4
312025-04-30T06:35:24.700ZINFOtest_metrics (store): pruning trees for generations newer than 0
322025-04-30T06:35:24.700ZINFOtest_metrics (store): pruning trees for generations older than 0 keep = 3
332025-04-30T06:35:24.700ZINFOtest_metrics (dns): DNS server bound to address kind = dns local_address = [::1]:64928
342025-04-30T06:35:24.701ZDEBGtest_metrics (http): registered endpoint kind = dns local_addr = [::1]:43730 method = GET path = /config versions = all
352025-04-30T06:35:24.701ZDEBGtest_metrics (http): registered endpoint kind = dns local_addr = [::1]:43730 method = PUT path = /config versions = all
362025-04-30T06:35:24.701ZINFOtest_metrics (http): listening kind = dns local_addr = [::1]:43730
372025-04-30T06:35:24.702ZDEBGtest_metrics (http): successfully registered DTrace USDT probes kind = dns local_addr = [::1]:43730
382025-04-30T06:35:24.702ZDEBGtest_metrics (init_with_steps): Running step start_nexus_internal
392025-04-30T06:35:24.702ZDEBGtest_metrics: Starting Nexus (internal API)
402025-04-30T06:35:24.703ZINFOtest_metrics: setting up nexus server
412025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = Action
422025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = AnyActor
432025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = AuthenticatedActor
442025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = BlueprintConfig
452025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = Database
462025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = DnsConfig
472025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = Fleet
482025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = Inventory
492025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = IpPoolList
502025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = ConsoleSessionList
512025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = DeviceAuthRequestList
522025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = SiloCertificateList
532025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = SiloIdentityProviderList
542025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = SiloUserList
552025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = TargetReleaseConfig
562025-04-30T06:35:24.706ZINFOtest_metrics (ServerContext): registering Oso class class = WebhookEventClassList
572025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = Project
582025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = Disk
592025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = Snapshot
602025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = ProjectImage
612025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = AffinityGroup
622025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = AntiAffinityGroup
632025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = Instance
642025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = IpPool
652025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = InstanceNetworkInterface
662025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = Vpc
672025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = VpcRouter
682025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = InternetGateway
692025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = InternetGatewayIpPool
702025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = InternetGatewayIpAddress
712025-04-30T06:35:24.707ZINFOtest_metrics (ServerContext): registering Oso class class = RouterRoute
722025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = VpcSubnet
732025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = FloatingIp
742025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = Image
752025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = SiloImage
762025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = AddressLot
772025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = Blueprint
782025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = LoopbackAddress
792025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = Certificate
802025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = ConsoleSession
812025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = DeviceAuthRequest
822025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = DeviceAccessToken
832025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = PhysicalDisk
842025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = Rack
852025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = RoleBuiltin
862025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = SshKey
872025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = Silo
882025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = SiloUser
892025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = SiloGroup
902025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = SupportBundle
912025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = IdentityProvider
922025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = SamlIdentityProvider
932025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = Sled
942025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = TufRepo
952025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = TufArtifact
962025-04-30T06:35:24.708ZINFOtest_metrics (ServerContext): registering Oso class class = WebhookEvent
972025-04-30T06:35:24.709ZINFOtest_metrics (ServerContext): registering Oso class class = WebhookReceiver
982025-04-30T06:35:24.709ZINFOtest_metrics (ServerContext): registering Oso class class = WebhookSecret
992025-04-30T06:35:24.709ZINFOtest_metrics (ServerContext): registering Oso class class = Zpool
1002025-04-30T06:35:24.709ZINFOtest_metrics (ServerContext): registering Oso class class = Service
1012025-04-30T06:35:24.709ZINFOtest_metrics (ServerContext): registering Oso class class = UserBuiltin
1022025-04-30T06:35:24.709ZINFOtest_metrics (ServerContext): full Oso configuration config = #\n# Oso configuration for Omicron\n# This file is augmented by generated snippets.\n#\n\n#\n# ACTOR TYPES AND BASIC RULES\n#\n\n# `AnyActor` includes both authenticated and unauthenticated users.\nactor AnyActor {}\n\n# An `AuthenticatedActor` has an identity in the system. All of our operations\n# today require that an actor be authenticated.\nactor AuthenticatedActor {}\n\n# For any resource, `actor` can perform action `action` on it if they're\n# authenticated and their role(s) give them the corresponding permission on that\n# resource.\nallow(actor: AnyActor, action: Action, resource) if\n actor.authenticated and\n has_permission(actor.authn_actor.unwrap(), action.to_perm(), resource);\n\n# Define role relationships\nhas_role(actor: AuthenticatedActor, role: String, resource: Resource)\n\tif resource.has_role(actor, role);\n\n#\n# ROLES AND PERMISSIONS IN THE FLEET/SILO/PROJECT HIERARCHY\n#\n# We define the following permissions for most resources in the system:\n#\n# - "create_child": required to create child resources (of any type)\n#\n# - "list_children": required to list child resources (of all types) of a\n# resource\n#\n# - "modify": required to modify or delete a resource\n#\n# - "read": required to read a resource\n#\n# We define the following predefined roles for only a few high-level resources:\n# the Fleet (see below), Silo, Organization, and Project. The specific roles\n# are oriented around intended use-cases:\n#\n# - "admin": has all permissions on the resource\n#\n# - "collaborator": has "read", "list_children", and "create_child", plus\n# the "admin" role for child resources. The idea is that if you're an\n# Organization Collaborator, you have full control over the Projects within\n# the Organization, but you cannot modify or delete the Organization itself.\n#\n# - "viewer": has "read" and "list_children" on a resource\n#\n# Below the Project level, permissions are granted via roles at the Project\n# level. For example, for someone to be able to create, modify, or delete any\n# Instances, they must be granted project.collaborator, which means they can\n# create, modify, or delete _all_ resources in the Project.\n#\n# The complete set of predefined roles:\n#\n# - fleet.admin (superuser for the whole system)\n# - fleet.collaborator (can manage Silos)\n# - fleet.viewer (can read most non-siloed resources in the system)\n# - silo.admin (superuser for the silo)\n# - silo.collaborator (can create and own Organizations)\n# - silo.viewer (can read most resources within the Silo)\n# - organization.admin (complete control over an organization)\n# - organization.collaborator (can manage Projects)\n# - organization.viewer (can read most resources within the Organization)\n# - project.admin (complete control over a Project)\n# - project.collaborator (can manage all resources within the Project)\n# - project.viewer (can read most resources within the Project)\n#\n# Outside the Silo/Organization/Project hierarchy, we (currently) treat most\n# resources as nested under Fleet or else a synthetic resource (see below). We\n# do not yet support role assignments on anything other than Fleet, Silo,\n# Organization, or Project.\n#\n\n# "Fleet" is a global singleton representing the whole system. The name comes\n# from the idea described in RFD 24, but it's not quite right. This probably\n# should be more like "Region" or "AvailabilityZone". The precise boundaries\n# have not yet been figured out.\nresource Fleet {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\troles = [\n\t # Roles that can be attached by users\n\t "admin",\n\t "collaborator",\n\t "viewer",\n\n\t # Internal-only roles\n\t "external-authenticator"\n\t];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n}\n\n# For fleets specifically, roles can be conferred by roles on the user's Silo.\nhas_role(actor: AuthenticatedActor, role: String, _: Fleet) if\n\tsilo_role in actor.confers_fleet_role(role) and\n\thas_role(actor, silo_role, actor.silo.unwrap());\n\nresource Silo {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Permissions implied by roles on this resource's parent (Fleet). Fleet\n\t# privileges allow a user to see and potentially administer the Silo,\n\t# but they do not give anyone permission to look at anything inside the\n\t# Silo. To achieve this, we use permission rules here. (If we granted\n\t# Fleet administrators _roles_ on the Silo, then those would cascade\n\t# into the Silo as well.)\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "collaborator" on "parent_fleet";\n\n\t# external authenticator has to create silo users\n\t"list_children" if "external-authenticator" on "parent_fleet";\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", silo: Silo)\n\tif silo.fleet = fleet;\n\n# As a special case, all authenticated users can read their own Silo. That's\n# not quite the same as having the "viewer" role. For example, they cannot list\n# Organizations in the Silo.\n#\n# One reason this is necessary is because if an unprivileged user tries to\n# create an Organization using "POST /organizations", they should get back a 403\n# (which implies they're able to see /organizations, which is essentially seeing\n# the Silo itself) rather than a 404. This behavior isn't a hard constraint\n# (i.e., you could reasonably get a 404 for an API you're not allowed to call).\n# Nor is the implementation (i.e., we could special-case this endpoint somehow).\n# But granting this permission is the simplest way to keep this endpoint's\n# behavior consistent with the rest of the API.\n#\n# This rule is also used to determine if a user can list the identity providers\n# in the Silo (which they should be able to), since that's predicated on being\n# able to read the Silo.\n#\n# It's unclear what else would break if users couldn't see their own Silo.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif silo in actor.silo;\n\nresource Project {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\troles = [ "admin", "collaborator", "viewer" ];\n\n\t# Roles implied by other roles on this resource\n\t"viewer" if "collaborator";\n\t"collaborator" if "admin";\n\n\t# Permissions granted directly by roles on this resource\n\t"list_children" if "viewer";\n\t"read" if "viewer";\n\t"create_child" if "collaborator";\n\t"modify" if "admin";\n\n\t# Roles implied by roles on this resource's parent (Silo)\n\trelations = { parent_silo: Silo };\n\t"admin" if "collaborator" on "parent_silo";\n\t"viewer" if "viewer" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", project: Project)\n\tif project.silo = silo;\n\n#\n# GENERAL RESOURCES OUTSIDE THE SILO/PROJECT HIERARCHY\n#\n# Many resources use snippets of Polar generated by the `authz_resource!` Rust\n# macro. Some resources require custom Polar code. Those appear here.\n#\n\nresource Certificate {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Fleet-level and silo-level roles both grant privileges on certificates.\n\t"read" if "admin" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", certificate: Certificate)\n\tif certificate.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", certificate: Certificate)\n\tif certificate.silo.fleet = fleet;\n\nresource SiloUser {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\t# Fleet and Silo administrators can manage a Silo's users. This is one\n\t# of the only areas of Silo configuration that Fleet Administrators have\n\t# permissions on.\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "read" on "parent_fleet";\n\t"read" if "read" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", user: SiloUser)\n\tif user.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", user: SiloUser)\n\tif user.silo.fleet = fleet;\n\n# authenticated actors have all permissions on themselves\nhas_permission(actor: AuthenticatedActor, _perm: String, silo_user: SiloUser)\n if actor.equals_silo_user(silo_user);\n\nhas_permission(actor: AuthenticatedActor, "read", silo_user: SiloUser)\n if silo_user.silo in actor.silo;\n\nresource SiloGroup {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "read",\n\t "create_child",\n\t];\n\n\trelations = { parent_silo: Silo };\n\t"list_children" if "read" on "parent_silo";\n\t"read" if "read" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n}\nhas_relation(silo: Silo, "parent_silo", group: SiloGroup)\n\tif group.silo = silo;\n\nresource SshKey {\n\tpermissions = [ "read", "modify" ];\n\trelations = { silo_user: SiloUser };\n\n\t"read" if "read" on "silo_user";\n\t"modify" if "modify" on "silo_user";\n}\nhas_relation(user: SiloUser, "silo_user", ssh_key: SshKey)\n\tif ssh_key.silo_user = user;\n\nresource IdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", identity_provider: IdentityProvider)\n\tif identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: IdentityProvider)\n\tif collection.silo.fleet = fleet;\n\nresource SamlIdentityProvider {\n\tpermissions = [\n\t "read",\n\t "modify",\n\t "create_child",\n\t "list_children",\n\t];\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Silo-level roles grant privileges on identity providers.\n\t"read" if "viewer" on "parent_silo";\n\t"list_children" if "viewer" on "parent_silo";\n\t"modify" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_silo";\n\n\t# Fleet-level roles also grant privileges on identity providers.\n\t"read" if "viewer" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", saml_identity_provider: SamlIdentityProvider)\n\tif saml_identity_provider.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SamlIdentityProvider)\n\tif collection.silo.fleet = fleet;\n\n#\n# SYNTHETIC RESOURCES OUTSIDE THE SILO HIERARCHY\n#\n# The resources here do not correspond to anything that appears explicitly in\n# the API or is stored in the database. These are used either at the top level\n# of the API path (e.g., "/v1/system/ip-pools") or as an implementation detail of the system\n# (in the case of console sessions and "Database"). The policies are\n# either statically-defined in this file or driven by role assignments on the\n# Fleet. None of these resources defines their own roles.\n#\n\n# Describes the policy for reading and modifying DNS configuration\n# (both internal and external)\nresource DnsConfig {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t# "external-authenticator" requires these permissions because that's the\n\t# context that Nexus uses when creating and deleting Silos. These\n\t# operations necessarily need to read and modify DNS configuration.\n\t"read" if "external-authenticator" on "parent_fleet";\n\t"modify" if "external-authenticator" on "parent_fleet";\n\t# "admin" on the parent fleet also gets these permissions, primarily for\n\t# the test suite.\n\t"read" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", dns_config: DnsConfig)\n\tif dns_config.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource BlueprintConfig {\n\tpermissions = [\n\t "list_children", # list blueprints\n\t "create_child", # create blueprint\n\t "read", # read the current target\n\t "modify", # change the current target\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "admin" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n\t"list_children" if "viewer" on "parent_fleet";\n\t"read" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", list: BlueprintConfig)\n\tif list.fleet = fleet;\n\n# Describes the policy for accessing blueprints\nresource TargetReleaseConfig {\n\tpermissions = [\n\t "read", # read the current target release\n\t "modify", # change the current target release\n\t];\n\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", resource: TargetReleaseConfig)\n\tif resource.fleet = fleet;\n\n# Describes the policy for reading and modifying low-level inventory\nresource Inventory {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_fleet: Fleet };\n\t"read" if "viewer" on "parent_fleet";\n\t"modify" if "admin" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", inventory: Inventory)\n\tif inventory.fleet = fleet;\n\n# Describes the policy for accessing "/v1/system/ip-pools" in the API\nresource IpPoolList {\n\tpermissions = [\n\t "list_children",\n\t "modify",\n\t "create_child",\n\t];\n\n\t# Fleet Administrators can create or modify the IP Pools list.\n\trelations = { parent_fleet: Fleet };\n\t"modify" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n\n\t# Fleet Viewers can list IP Pools\n\t"list_children" if "viewer" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", ip_pool_list: IpPoolList)\n\tif ip_pool_list.fleet = fleet;\n\n# Any authenticated user can create a child of a provided IP Pool.\n# This is necessary to use the pools when provisioning instances.\nhas_permission(actor: AuthenticatedActor, "create_child", ip_pool: IpPool)\n\tif silo in actor.silo and silo.fleet = ip_pool.fleet;\n\n# Describes the policy for creating and managing web console sessions.\nresource ConsoleSessionList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: ConsoleSessionList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing device authorization requests.\nresource DeviceAuthRequestList {\n\tpermissions = [ "create_child" ];\n\trelations = { parent_fleet: Fleet };\n\t"create_child" if "external-authenticator" on "parent_fleet";\n}\nhas_relation(fleet: Fleet, "parent_fleet", collection: DeviceAuthRequestList)\n\tif collection.fleet = fleet;\n\n# Describes the policy for creating and managing Silo certificates\nresource SiloCertificateList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Both Fleet and Silo administrators can see and modify the Silo's\n\t# certificates.\n\t"list_children" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloCertificateList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloCertificateList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo identity providers\nresource SiloIdentityProviderList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the identity providers in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's identity provider\n\t# configuration. This is one of the only areas of Silo configuration\n\t# that Fleet Administrators have permissions on. This is also one of\n\t# the only cases where we need to look two levels up the hierarchy to\n\t# see if somebody has the right permission. For most other things,\n\t# permissions cascade down the hierarchy so we only need to look at the\n\t# parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloIdentityProviderList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloIdentityProviderList)\n\tif collection.silo.fleet = fleet;\n\n# Describes the policy for creating and managing Silo users (mostly intended for\n# API-managed users)\nresource SiloUserList {\n\tpermissions = [ "list_children", "create_child" ];\n\n\trelations = { parent_silo: Silo, parent_fleet: Fleet };\n\n\t# Everyone who can read the Silo (which includes all the users in the\n\t# Silo) can see the users in it.\n\t"list_children" if "read" on "parent_silo";\n\n\t# Fleet and Silo administrators can manage the Silo's users. This is\n\t# one of the only areas of Silo configuration that Fleet Administrators\n\t# have permissions on. This is also one of the few cases (so far) where\n\t# we need to look two levels up the hierarchy to see if somebody has the\n\t# right permission. For most other things, permissions cascade down the\n\t# hierarchy so we only need to look at the parent.\n\t"create_child" if "admin" on "parent_silo";\n\t"list_children" if "admin" on "parent_fleet";\n\t"create_child" if "admin" on "parent_fleet";\n}\nhas_relation(silo: Silo, "parent_silo", collection: SiloUserList)\n\tif collection.silo = silo;\nhas_relation(fleet: Fleet, "parent_fleet", collection: SiloUserList)\n\tif collection.silo.fleet = fleet;\n\n# These rules grants the external authenticator role the permissions it needs to\n# read silo users and modify their sessions. This is necessary for login to\n# work.\nhas_permission(actor: AuthenticatedActor, "read", silo: Silo)\n\tif has_role(actor, "external-authenticator", silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", user: SiloUser)\n\tif has_role(actor, "external-authenticator", user.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "read", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", group: SiloGroup)\n\tif has_role(actor, "external-authenticator", group.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\nhas_permission(actor: AuthenticatedActor, "modify", session: ConsoleSession)\n\tif has_role(actor, "external-authenticator", session.fleet);\n\n# All authenticated users can read and delete device authn requests because\n# by necessity these operations happen before we've figured out what user (or\n# even Silo) the device auth is associated with. Any user can claim a device\n# auth request with the right user code (that's how it works) -- it's the user\n# code and associated logic that prevents unauthorized access here.\nhas_permission(_actor: AuthenticatedActor, "read", _device_auth: DeviceAuthRequest);\nhas_permission(_actor: AuthenticatedActor, "modify", _device_auth: DeviceAuthRequest);\n\nhas_permission(actor: AuthenticatedActor, "read", device_token: DeviceAccessToken)\n\tif has_role(actor, "external-authenticator", device_token.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", identity_provider: IdentityProvider)\n\tif has_role(actor, "external-authenticator", identity_provider.silo.fleet);\n\nhas_permission(actor: AuthenticatedActor, "read", saml_identity_provider: SamlIdentityProvider)\n\tif has_role(actor, "external-authenticator", saml_identity_provider.silo.fleet);\n\n# Describes the policy for who can access the internal database.\nresource Database {\n\tpermissions = [\n\t # "query" is required to perform any query against the database,\n\t # whether a read or write query. This is checked when an operation\n\t # checks out a database connection from the connection pool.\n\t #\n\t # Any authenticated user gets this permission. There's generally\n\t # some other authz check involved in the database query. For\n\t # example, if you're querying the database to "read" a "Project", we\n\t # should also be checking that. So why do we do this at all? It's\n\t # a belt-and-suspenders measure so that if we somehow introduced an\n\t # unauthenticated code path that hits the database, it cannot be\n\t # used to DoS the database because we won't allow the operation to\n\t # make the query. (As long as the code path _is_ authenticated, we\n\t # can use throttling mechanisms to prevent DoS.)\n\t "query",\n\n\t # "modify" is required to populate database data that's delivered\n\t # with the system. It should also be required for schema changes,\n\t # when we support those. This is separate from "query" so that we\n\t # cannot accidentally invoke these code paths from API calls and\n\t # other general functions.\n\t "modify"\n\t];\n}\n\n# All authenticated users have the "query" permission on the database.\nhas_permission(_actor: AuthenticatedActor, "query", _resource: Database);\n\n# The "db-init" user is the only one with the "modify" permission.\nhas_permission(USER_DB_INIT: AuthenticatedActor, "modify", _resource: Database);\nhas_permission(USER_DB_INIT: AuthenticatedActor, "create_child", _resource: IpPoolList);\n# It also has "admin" on the internal silo to populate it with built-in resources.\n# TODO-completeness: actually limit to just internal silo and not all silos\nhas_role(USER_DB_INIT: AuthenticatedActor, "admin", _silo: Silo);\n\n# Allow the internal API admin permissions on all silos.\nhas_role(USER_INTERNAL_API: AuthenticatedActor, "admin", _silo: Silo);\n\nresource WebhookSecret {\n\tpermissions = [ "read", "modify" ];\n\trelations = { parent_webhook_receiver: WebhookReceiver };\n\n\t"read" if "read" on "parent_webhook_receiver";\n\t"modify" if "modify" on "parent_webhook_receiver";\n}\n\nhas_relation(rx: WebhookReceiver, "parent_webhook_receiver", secret: WebhookSecret)\n\tif secret.webhook_receiver = rx;\n\nresource WebhookEventClassList {\n\tpermissions = [ "list_children" ];\n\trelations = { parent_fleet: Fleet };\n\n\t"list_children" if "viewer" on "parent_fleet";\n}\n\nhas_relation(fleet: Fleet, "parent_fleet", collection: WebhookEventClassList)\n\tif collection.fleet = fleet;\n\n\n\n resource Disk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Disk)\n if child.project = parent;\n \n\n resource Snapshot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Snapshot)\n if child.project = parent;\n \n\n resource ProjectImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: ProjectImage)\n if child.project = parent;\n \n\n resource AffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AffinityGroup)\n if child.project = parent;\n \n\n resource AntiAffinityGroup {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: AntiAffinityGroup)\n if child.project = parent;\n \n\n resource Instance {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Instance)\n if child.project = parent;\n \n\n resource IpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: IpPool)\n if child.fleet = fleet;\n \n\n resource InstanceNetworkInterface {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Instance\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InstanceNetworkInterface)\n if has_relation(project, "containing_project", child.instance);\n\n has_relation(parent: Instance, "parent", child: InstanceNetworkInterface)\n if child.instance = parent;\n \n\n resource Vpc {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: Vpc)\n if child.project = parent;\n \n\n resource VpcRouter {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcRouter)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcRouter)\n if child.vpc = parent;\n \n\n resource InternetGateway {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGateway)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: InternetGateway)\n if child.vpc = parent;\n \n\n resource InternetGatewayIpPool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpPool)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpPool)\n if child.internet_gateway = parent;\n \n\n resource InternetGatewayIpAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: InternetGateway\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: InternetGatewayIpAddress)\n if has_relation(project, "containing_project", child.internet_gateway);\n\n has_relation(parent: InternetGateway, "parent", child: InternetGatewayIpAddress)\n if child.internet_gateway = parent;\n \n\n resource RouterRoute {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: VpcRouter\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: RouterRoute)\n if has_relation(project, "containing_project", child.vpc_router);\n\n has_relation(parent: VpcRouter, "parent", child: RouterRoute)\n if child.vpc_router = parent;\n \n\n resource VpcSubnet {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = {\n containing_project: Project,\n parent: Vpc\n };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(project: Project, "containing_project", child: VpcSubnet)\n if has_relation(project, "containing_project", child.vpc);\n\n has_relation(parent: Vpc, "parent", child: VpcSubnet)\n if child.vpc = parent;\n \n\n resource FloatingIp {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_project: Project };\n "list_children" if "viewer" on "containing_project";\n "read" if "viewer" on "containing_project";\n "modify" if "collaborator" on "containing_project";\n "create_child" if "collaborator" on "containing_project";\n }\n\n has_relation(parent: Project, "containing_project", child: FloatingIp)\n if child.project = parent;\n \n\n resource Image {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: Image)\n if child.silo = parent;\n \n\n resource SiloImage {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n\n relations = { containing_silo: Silo };\n "list_children" if "viewer" on "containing_silo";\n "read" if "viewer" on "containing_silo";\n "modify" if "collaborator" on "containing_silo";\n "create_child" if "collaborator" on "containing_silo";\n }\n\n has_relation(parent: Silo, "containing_silo", child: SiloImage)\n if child.silo = parent;\n \n\n resource AddressLot {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: AddressLot)\n if child.fleet = fleet;\n \n\n resource Blueprint {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Blueprint)\n if child.fleet = fleet;\n \n\n resource LoopbackAddress {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: LoopbackAddress)\n if child.fleet = fleet;\n \n\n\n resource ConsoleSession {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: ConsoleSession)\n if child.fleet = fleet;\n \n\n resource DeviceAuthRequest {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAuthRequest)\n if child.fleet = fleet;\n \n\n resource DeviceAccessToken {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: DeviceAccessToken)\n if child.fleet = fleet;\n \n\n resource PhysicalDisk {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: PhysicalDisk)\n if child.fleet = fleet;\n \n\n resource Rack {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Rack)\n if child.fleet = fleet;\n \n\n resource RoleBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: RoleBuiltin)\n if child.fleet = fleet;\n \n\n\n\n\n\n resource SupportBundle {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: SupportBundle)\n if child.fleet = fleet;\n \n\n\n\n resource Sled {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Sled)\n if child.fleet = fleet;\n \n\n resource TufRepo {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufRepo)\n if child.fleet = fleet;\n \n\n resource TufArtifact {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: TufArtifact)\n if child.fleet = fleet;\n \n\n resource WebhookEvent {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookEvent)\n if child.fleet = fleet;\n \n\n resource WebhookReceiver {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: WebhookReceiver)\n if child.fleet = fleet;\n \n\n\n resource Zpool {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Zpool)\n if child.fleet = fleet;\n \n\n resource Service {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: Service)\n if child.fleet = fleet;\n \n\n resource UserBuiltin {\n permissions = [\n "list_children",\n "modify",\n "read",\n "create_child",\n ];\n \n relations = { parent_fleet: Fleet };\n "list_children" if "viewer" on "parent_fleet";\n "read" if "viewer" on "parent_fleet";\n "modify" if "admin" on "parent_fleet";\n "create_child" if "admin" on "parent_fleet";\n }\n has_relation(fleet: Fleet, "parent_fleet", child: UserBuiltin)\n if child.fleet = fleet;\n
1032025-04-30T06:35:24.732ZINFOtest_metrics (ServerContext): Setting up resolver using DNS address: [::1]:53219
1042025-04-30T06:35:24.732ZINFOtest_metrics (DnsResolver): new DNS resolver addresses = [[::1]:53219]
1052025-04-30T06:35:24.733ZINFOtest_metrics (ServerContext): Setting up qorb database pool from a single host url = PostgresConfigWithUrl {\n url_raw: "postgresql://root@[::1]:57004/omicron?sslmode=disable",\n config: Config {\n user: Some(\n "root",\n ),\n password: None,\n dbname: Some(\n "omicron",\n ),\n options: None,\n application_name: None,\n ssl_mode: Disable,\n host: [\n Tcp(\n "::1",\n ),\n ],\n hostaddr: [],\n port: [\n 57004,\n ],\n connect_timeout: None,\n tcp_user_timeout: None,\n keepalives: true,\n keepalives_idle: 7200s,\n keepalives_interval: None,\n keepalives_retries: None,\n target_session_attrs: Any,\n channel_binding: Prefer,\n load_balance_hosts: Disable,\n },\n}
1062025-04-30T06:35:24.735ZDEBGtest_metrics (ServerContext): registered USDT probes
1072025-04-30T06:35:24.824ZINFOtest_metrics (nexus): Database schema version is up to date desired_version = 140.0.0 found_version = 140.0.0
1082025-04-30T06:35:25.046ZDEBGtest_metrics (clickhouse-client): registered USDT probes id = 632a8471-c10c-4502-81cd-5651fb1eec17
1092025-04-30T06:35:25.202ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /bgtasks versions = all
1102025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /bgtasks/activate versions = all
1112025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /bgtasks/view/{bgtask_name} versions = all
1122025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /clickhouse/policy versions = all
1132025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /clickhouse/policy versions = all
1142025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stop-request versions = all
1152025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /crucible/0/upstairs/{upstairs_id}/downstairs/{downstairs_id}/stopped versions = all
1162025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair/{repair_id}/progress versions = all
1172025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-finish versions = all
1182025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /crucible/0/upstairs/{upstairs_id}/repair-start versions = all
1192025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /demo-saga versions = all
1202025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /demo-saga/{demo_saga_id}/complete versions = all
1212025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /deployment/blueprints/all versions = all
1222025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = DELETE path = /deployment/blueprints/all/{blueprint_id} versions = all
1232025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /deployment/blueprints/all/{blueprint_id} versions = all
1242025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /deployment/blueprints/import versions = all
1252025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /deployment/blueprints/regenerate versions = all
1262025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /deployment/blueprints/target versions = all
1272025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /deployment/blueprints/target versions = all
1282025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = PUT path = /deployment/blueprints/target/enabled versions = all
1292025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /disk/{disk_id}/remove-read-only-parent versions = all
1302025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = PUT path = /disks/{disk_id} versions = all
1312025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /experimental/v1/system/support-bundles versions = all
1322025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /experimental/v1/system/support-bundles versions = all
1332025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1342025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
1352025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1362025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
1372025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1382025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
1392025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
1402025-04-30T06:35:25.203ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /instances/{instance_id}/migrate versions = all
1412025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /metrics/collectors versions = all
1422025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /metrics/collectors/{collector_id}/producers versions = all
1432025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /metrics/producers versions = all
1442025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /mgs-updates versions = all
1452025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /nat/ipv4/changeset/{from_gen} versions = all
1462025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /oximeter/read-policy versions = all
1472025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /oximeter/read-policy versions = all
1482025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /physical-disk/expunge versions = all
1492025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /probes/{sled} versions = all
1502025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = PUT path = /racks/{rack_id}/initialization-complete versions = all
1512025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /sagas versions = all
1522025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /sagas/{saga_id} versions = all
1532025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /sled-agents/{sled_id} versions = all
1542025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /sled-agents/{sled_id} versions = all
1552025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /sled-agents/{sled_id}/firewall-rules-update versions = all
1562025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /sleds/add versions = all
1572025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /sleds/expunge versions = all
1582025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /sleds/uninitialized versions = all
1592025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = PUT path = /switch/{switch_id} versions = all
1602025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = GET path = /v1/ping versions = all
1612025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = PUT path = /vmms/{propolis_id} versions = all
1622025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): registered endpoint local_addr = 127.0.0.1:49095 method = POST path = /volume/{volume_id}/remove-read-only-parent versions = all
1632025-04-30T06:35:25.204ZINFOtest_metrics (dropshot_internal): listening local_addr = 127.0.0.1:49095
1642025-04-30T06:35:25.204ZDEBGtest_metrics (dropshot_internal): successfully registered DTrace USDT probes local_addr = 127.0.0.1:49095
1652025-04-30T06:35:25.204ZINFOtest_metrics (SEC): SEC running sec_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
1662025-04-30T06:35:25.206ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1672025-04-30T06:35:25.206ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1682025-04-30T06:35:25.212ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1692025-04-30T06:35:25.217ZDEBGtest_metrics (DataLoader): attempting to create built-in users
1702025-04-30T06:35:25.219ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1712025-04-30T06:35:25.219ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1722025-04-30T06:35:25.219ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1732025-04-30T06:35:25.220ZINFOtest_metrics (MgsUpdateDriver): starting MgsUpdateDriver
1742025-04-30T06:35:25.233ZTRACtest_metrics (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: ed 53 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 .S..........._cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:64297 req_id = 759fa385-7a0e-4afd-b27c-bc87e1a05d0c
1752025-04-30T06:35:25.233ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60755,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64297 req_id = 759fa385-7a0e-4afd-b27c-bc87e1a05d0c
1762025-04-30T06:35:25.235ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 60755, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_crucible-pantry._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 16, 95, 99, 114, 117, 99, 105, 98, 108, 101, 45, 112, 97, 110, 116, 114, 121, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_crucible-pantry._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64297 req_id = 759fa385-7a0e-4afd-b27c-bc87e1a05d0c
1772025-04-30T06:35:25.265ZINFOtest_metrics (DataLoader): created 5 built-in users
1782025-04-30T06:35:25.280ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1792025-04-30T06:35:25.280ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1802025-04-30T06:35:25.301ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1812025-04-30T06:35:25.301ZDEBGtest_metrics (DataLoader): attempting to create built-in roles
1822025-04-30T06:35:25.301ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1832025-04-30T06:35:25.301ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1842025-04-30T06:35:25.302ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1852025-04-30T06:35:25.380ZINFOtest_metrics (DataLoader): created 10 built-in roles
1862025-04-30T06:35:25.380ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1872025-04-30T06:35:25.380ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1882025-04-30T06:35:25.383ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1892025-04-30T06:35:25.383ZDEBGtest_metrics (DataLoader): attempting to create built-in role assignments
1902025-04-30T06:35:25.383ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1912025-04-30T06:35:25.383ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1922025-04-30T06:35:25.384ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1932025-04-30T06:35:25.673ZINFOtest_metrics (DataLoader): created 4 built-in role assignments
1942025-04-30T06:35:25.673ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1952025-04-30T06:35:25.673ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
1962025-04-30T06:35:25.675ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
1972025-04-30T06:35:25.675ZDEBGtest_metrics (DataLoader): attempting to create built-in silos
1982025-04-30T06:35:25.675ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
1992025-04-30T06:35:25.675ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2002025-04-30T06:35:25.676ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2012025-04-30T06:35:25.786ZINFOtest_metrics (DataLoader): created 2 built-in silos
2022025-04-30T06:35:25.786ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2032025-04-30T06:35:25.786ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2042025-04-30T06:35:25.787ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2052025-04-30T06:35:25.801ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2062025-04-30T06:35:25.801ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2072025-04-30T06:35:25.803ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2082025-04-30T06:35:25.803ZDEBGtest_metrics (DataLoader): attempting to create built-in projects
2092025-04-30T06:35:25.803ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2102025-04-30T06:35:25.803ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2112025-04-30T06:35:25.804ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2122025-04-30T06:35:25.825ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2132025-04-30T06:35:25.825ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2142025-04-30T06:35:25.825ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2152025-04-30T06:35:25.825ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2162025-04-30T06:35:25.826ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2172025-04-30T06:35:25.910ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2182025-04-30T06:35:25.910ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2192025-04-30T06:35:25.910ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2202025-04-30T06:35:25.912ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2212025-04-30T06:35:25.920ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2222025-04-30T06:35:25.923ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2232025-04-30T06:35:25.923ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }
2242025-04-30T06:35:25.923ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2252025-04-30T06:35:25.923ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2262025-04-30T06:35:25.923ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2272025-04-30T06:35:25.924ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2282025-04-30T06:35:25.927ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2292025-04-30T06:35:25.928ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2302025-04-30T06:35:25.928ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2312025-04-30T06:35:25.928ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2322025-04-30T06:35:25.941ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2332025-04-30T06:35:25.945ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) } result = Ok(())
2342025-04-30T06:35:25.945ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2352025-04-30T06:35:25.945ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2362025-04-30T06:35:25.946ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2372025-04-30T06:35:25.985ZINFOtest_metrics (DataLoader): created built-in services project
2382025-04-30T06:35:25.985ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2392025-04-30T06:35:25.985ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2402025-04-30T06:35:25.987ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2412025-04-30T06:35:25.988ZDEBGtest_metrics (DataLoader): attempting to create built-in VPCs
2422025-04-30T06:35:25.988ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2432025-04-30T06:35:25.988ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2442025-04-30T06:35:25.988ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2452025-04-30T06:35:25.991ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2462025-04-30T06:35:25.991ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2472025-04-30T06:35:25.991ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2482025-04-30T06:35:25.994ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2492025-04-30T06:35:25.994ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2502025-04-30T06:35:25.994ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2512025-04-30T06:35:25.994ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2522025-04-30T06:35:25.994ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2532025-04-30T06:35:25.997ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2542025-04-30T06:35:25.997ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2552025-04-30T06:35:25.997ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2562025-04-30T06:35:25.997ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2572025-04-30T06:35:26.000ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2582025-04-30T06:35:26.000ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2592025-04-30T06:35:26.000ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2602025-04-30T06:35:26.001ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2612025-04-30T06:35:26.063ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2622025-04-30T06:35:26.068ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2632025-04-30T06:35:26.068ZTRACtest_metrics (DataLoader): successful lookup of siloed resource "Project" using built-in user
2642025-04-30T06:35:26.068ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }
2652025-04-30T06:35:26.068ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2662025-04-30T06:35:26.068ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2672025-04-30T06:35:26.068ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2682025-04-30T06:35:26.068ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2692025-04-30T06:35:26.071ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2702025-04-30T06:35:26.071ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2712025-04-30T06:35:26.071ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2722025-04-30T06:35:26.071ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2732025-04-30T06:35:26.074ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2742025-04-30T06:35:26.074ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2752025-04-30T06:35:26.074ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2762025-04-30T06:35:26.087ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2772025-04-30T06:35:26.092ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2782025-04-30T06:35:26.096ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) } result = Ok(())
2792025-04-30T06:35:26.096ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2802025-04-30T06:35:26.096ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2812025-04-30T06:35:26.097ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2822025-04-30T06:35:26.108ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
2832025-04-30T06:35:26.108ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
2842025-04-30T06:35:26.108ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2852025-04-30T06:35:26.108ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2862025-04-30T06:35:26.109ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2872025-04-30T06:35:26.111ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
2882025-04-30T06:35:26.111ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2892025-04-30T06:35:26.112ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2902025-04-30T06:35:26.112ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2912025-04-30T06:35:26.114ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
2922025-04-30T06:35:26.114ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2932025-04-30T06:35:26.114ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2942025-04-30T06:35:26.116ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
2952025-04-30T06:35:26.119ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2962025-04-30T06:35:26.124ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
2972025-04-30T06:35:26.125ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
2982025-04-30T06:35:26.125ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
2992025-04-30T06:35:26.125ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3002025-04-30T06:35:26.139ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3012025-04-30T06:35:26.139ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3022025-04-30T06:35:26.139ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3032025-04-30T06:35:26.147ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3042025-04-30T06:35:26.148ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3052025-04-30T06:35:26.148ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3062025-04-30T06:35:26.169ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }
3072025-04-30T06:35:26.169ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3082025-04-30T06:35:26.169ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3092025-04-30T06:35:26.169ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3102025-04-30T06:35:26.170ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3112025-04-30T06:35:26.172ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3122025-04-30T06:35:26.172ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3132025-04-30T06:35:26.172ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3142025-04-30T06:35:26.173ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3152025-04-30T06:35:26.175ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3162025-04-30T06:35:26.175ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3172025-04-30T06:35:26.175ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3182025-04-30T06:35:26.176ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3192025-04-30T06:35:26.177ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3202025-04-30T06:35:26.184ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") } result = Ok(())
3212025-04-30T06:35:26.184ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3222025-04-30T06:35:26.184ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3232025-04-30T06:35:26.184ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3242025-04-30T06:35:26.259ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3252025-04-30T06:35:26.259ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3262025-04-30T06:35:26.259ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3272025-04-30T06:35:26.259ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3282025-04-30T06:35:26.260ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3292025-04-30T06:35:26.262ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3302025-04-30T06:35:26.262ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3312025-04-30T06:35:26.262ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3322025-04-30T06:35:26.263ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3332025-04-30T06:35:26.265ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3342025-04-30T06:35:26.265ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3352025-04-30T06:35:26.265ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3362025-04-30T06:35:26.265ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3372025-04-30T06:35:26.267ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3382025-04-30T06:35:26.274ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3392025-04-30T06:35:26.274ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3402025-04-30T06:35:26.274ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3412025-04-30T06:35:26.274ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3422025-04-30T06:35:26.341ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
3432025-04-30T06:35:26.341ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3442025-04-30T06:35:26.341ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3452025-04-30T06:35:26.341ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3462025-04-30T06:35:26.342ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3472025-04-30T06:35:26.351ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3482025-04-30T06:35:26.351ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3492025-04-30T06:35:26.351ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3502025-04-30T06:35:26.352ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3512025-04-30T06:35:26.357ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3522025-04-30T06:35:26.388ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3532025-04-30T06:35:26.388ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3542025-04-30T06:35:26.388ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3552025-04-30T06:35:26.391ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3562025-04-30T06:35:26.397ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
3572025-04-30T06:35:26.398ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3582025-04-30T06:35:26.398ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3592025-04-30T06:35:26.398ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3602025-04-30T06:35:26.406ZDEBGtest_metrics (DataLoader): attempting to create built-in VPC firewall rules
3612025-04-30T06:35:26.406ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3622025-04-30T06:35:26.406ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3632025-04-30T06:35:26.407ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3642025-04-30T06:35:26.410ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3652025-04-30T06:35:26.410ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3662025-04-30T06:35:26.410ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3672025-04-30T06:35:26.414ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3682025-04-30T06:35:26.414ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3692025-04-30T06:35:26.415ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3702025-04-30T06:35:26.419ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3712025-04-30T06:35:26.419ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3722025-04-30T06:35:26.419ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3732025-04-30T06:35:26.419ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3742025-04-30T06:35:26.419ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3752025-04-30T06:35:26.423ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3762025-04-30T06:35:26.423ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3772025-04-30T06:35:26.423ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3782025-04-30T06:35:26.425ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3792025-04-30T06:35:26.428ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3802025-04-30T06:35:26.428ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3812025-04-30T06:35:26.428ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3822025-04-30T06:35:26.429ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3832025-04-30T06:35:26.431ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3842025-04-30T06:35:26.440ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
3852025-04-30T06:35:26.440ZTRACtest_metrics (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
3862025-04-30T06:35:26.440ZTRACtest_metrics (DataLoader): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
3872025-04-30T06:35:26.441ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
3882025-04-30T06:35:26.441ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3892025-04-30T06:35:26.441ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3902025-04-30T06:35:26.441ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3912025-04-30T06:35:26.446ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
3922025-04-30T06:35:26.446ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3932025-04-30T06:35:26.446ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3942025-04-30T06:35:26.447ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3952025-04-30T06:35:26.449ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
3962025-04-30T06:35:26.450ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
3972025-04-30T06:35:26.450ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
3982025-04-30T06:35:26.450ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
3992025-04-30T06:35:26.453ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4002025-04-30T06:35:26.465ZDEBGtest_metrics (DataLoader): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4012025-04-30T06:35:26.466ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4022025-04-30T06:35:26.466ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4032025-04-30T06:35:26.466ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4042025-04-30T06:35:26.476ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4052025-04-30T06:35:26.477ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4062025-04-30T06:35:26.477ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4072025-04-30T06:35:26.477ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4082025-04-30T06:35:26.477ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4092025-04-30T06:35:26.480ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4102025-04-30T06:35:26.480ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4112025-04-30T06:35:26.480ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4122025-04-30T06:35:26.481ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4132025-04-30T06:35:26.483ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4142025-04-30T06:35:26.483ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4152025-04-30T06:35:26.483ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4162025-04-30T06:35:26.484ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4172025-04-30T06:35:26.486ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4182025-04-30T06:35:26.495ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4192025-04-30T06:35:26.495ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4202025-04-30T06:35:26.495ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4212025-04-30T06:35:26.495ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4222025-04-30T06:35:26.600ZINFOtest_metrics (DataLoader): created built-in VPC firewall rules
4232025-04-30T06:35:26.600ZDEBGtest_metrics (DataLoader): attempting to create built-in VPC Subnets
4242025-04-30T06:35:26.600ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4252025-04-30T06:35:26.600ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4262025-04-30T06:35:26.601ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4272025-04-30T06:35:26.606ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4282025-04-30T06:35:26.606ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4292025-04-30T06:35:26.607ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4302025-04-30T06:35:26.610ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4312025-04-30T06:35:26.610ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4322025-04-30T06:35:26.612ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4332025-04-30T06:35:26.618ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4342025-04-30T06:35:26.618ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4352025-04-30T06:35:26.618ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4362025-04-30T06:35:26.618ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4372025-04-30T06:35:26.621ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4382025-04-30T06:35:26.624ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4392025-04-30T06:35:26.624ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4402025-04-30T06:35:26.625ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4412025-04-30T06:35:26.626ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4422025-04-30T06:35:26.630ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4432025-04-30T06:35:26.630ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4442025-04-30T06:35:26.630ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4452025-04-30T06:35:26.631ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4462025-04-30T06:35:26.636ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4472025-04-30T06:35:26.645ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4482025-04-30T06:35:26.645ZTRACtest_metrics (DataLoader): successful lookup of siloed resource "Vpc" using built-in user
4492025-04-30T06:35:26.646ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4502025-04-30T06:35:26.646ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4512025-04-30T06:35:26.648ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4522025-04-30T06:35:26.658ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4532025-04-30T06:35:26.658ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4542025-04-30T06:35:26.658ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4552025-04-30T06:35:26.658ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4562025-04-30T06:35:26.659ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4572025-04-30T06:35:26.661ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4582025-04-30T06:35:26.661ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4592025-04-30T06:35:26.661ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4602025-04-30T06:35:26.662ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4612025-04-30T06:35:26.667ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4622025-04-30T06:35:26.667ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4632025-04-30T06:35:26.667ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4642025-04-30T06:35:26.668ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4652025-04-30T06:35:26.671ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4662025-04-30T06:35:26.677ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
4672025-04-30T06:35:26.685ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
4682025-04-30T06:35:26.685ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4692025-04-30T06:35:26.685ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4702025-04-30T06:35:26.685ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4712025-04-30T06:35:26.686ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4722025-04-30T06:35:26.688ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4732025-04-30T06:35:26.688ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4742025-04-30T06:35:26.688ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4752025-04-30T06:35:26.689ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4762025-04-30T06:35:26.694ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4772025-04-30T06:35:26.694ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4782025-04-30T06:35:26.695ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4792025-04-30T06:35:26.695ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4802025-04-30T06:35:26.699ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4812025-04-30T06:35:26.707ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
4822025-04-30T06:35:26.708ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4832025-04-30T06:35:26.708ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4842025-04-30T06:35:26.709ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4852025-04-30T06:35:26.803ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4862025-04-30T06:35:26.803ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4872025-04-30T06:35:26.804ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4882025-04-30T06:35:26.809ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
4892025-04-30T06:35:26.809ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
4902025-04-30T06:35:26.809ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4912025-04-30T06:35:26.809ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4922025-04-30T06:35:26.809ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4932025-04-30T06:35:26.811ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
4942025-04-30T06:35:26.811ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4952025-04-30T06:35:26.811ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
4962025-04-30T06:35:26.812ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
4972025-04-30T06:35:26.814ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
4982025-04-30T06:35:26.814ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
4992025-04-30T06:35:26.815ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5002025-04-30T06:35:26.815ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5012025-04-30T06:35:26.817ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5022025-04-30T06:35:26.824ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5032025-04-30T06:35:26.830ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5042025-04-30T06:35:26.830ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5052025-04-30T06:35:26.830ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5062025-04-30T06:35:26.830ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5072025-04-30T06:35:26.831ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5082025-04-30T06:35:26.833ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5092025-04-30T06:35:26.833ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5102025-04-30T06:35:26.833ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5112025-04-30T06:35:26.834ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5122025-04-30T06:35:26.838ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5132025-04-30T06:35:26.838ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5142025-04-30T06:35:26.838ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5152025-04-30T06:35:26.838ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5162025-04-30T06:35:26.843ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5172025-04-30T06:35:26.850ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5182025-04-30T06:35:26.850ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5192025-04-30T06:35:26.850ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5202025-04-30T06:35:26.851ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5212025-04-30T06:35:26.866ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5222025-04-30T06:35:26.866ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5232025-04-30T06:35:26.867ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5242025-04-30T06:35:26.871ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
5252025-04-30T06:35:26.871ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5262025-04-30T06:35:26.872ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5272025-04-30T06:35:26.872ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5282025-04-30T06:35:26.873ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5292025-04-30T06:35:26.883ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5302025-04-30T06:35:26.883ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5312025-04-30T06:35:26.883ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5322025-04-30T06:35:26.884ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5332025-04-30T06:35:26.890ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5342025-04-30T06:35:26.890ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5352025-04-30T06:35:26.890ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5362025-04-30T06:35:26.891ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5372025-04-30T06:35:26.898ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5382025-04-30T06:35:26.907ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
5392025-04-30T06:35:26.921ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) }
5402025-04-30T06:35:26.922ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
5412025-04-30T06:35:26.922ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5422025-04-30T06:35:26.922ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5432025-04-30T06:35:26.922ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5442025-04-30T06:35:26.948ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
5452025-04-30T06:35:26.948ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5462025-04-30T06:35:26.948ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5472025-04-30T06:35:26.949ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5482025-04-30T06:35:26.953ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5492025-04-30T06:35:26.953ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5502025-04-30T06:35:26.953ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5512025-04-30T06:35:26.953ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5522025-04-30T06:35:26.965ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5532025-04-30T06:35:26.974ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = VpcRouter { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-074c-4000-8000-000000000001, lookup_type: ById(001de000-074c-4000-8000-000000000001) } result = Ok(())
5542025-04-30T06:35:26.974ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5552025-04-30T06:35:26.974ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5562025-04-30T06:35:26.974ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5572025-04-30T06:35:26.983ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5582025-04-30T06:35:26.989ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5592025-04-30T06:35:26.990ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5602025-04-30T06:35:26.993ZINFOtest_metrics (DataLoader): created built-in services vpc subnets
5612025-04-30T06:35:26.993ZINFOtest_metrics (DataLoader): created built-in services vpc
5622025-04-30T06:35:26.993ZTRACtest_metrics (DataLoader): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5632025-04-30T06:35:26.993ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5642025-04-30T06:35:26.996ZDEBGtest_metrics (DataLoader): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5652025-04-30T06:35:26.996ZDEBGtest_metrics (DataLoader): attempting to create silo users
5662025-04-30T06:35:26.996ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5672025-04-30T06:35:26.996ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5682025-04-30T06:35:26.997ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5692025-04-30T06:35:27.005ZINFOtest_metrics (DataLoader): created 2 silo users
5702025-04-30T06:35:27.005ZDEBGtest_metrics (DataLoader): attempting to create silo user role assignments
5712025-04-30T06:35:27.005ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5722025-04-30T06:35:27.005ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5732025-04-30T06:35:27.007ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5742025-04-30T06:35:27.013ZINFOtest_metrics (DataLoader): created 2 silo user role assignments
5752025-04-30T06:35:27.013ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5762025-04-30T06:35:27.013ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5772025-04-30T06:35:27.014ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5782025-04-30T06:35:27.028ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5792025-04-30T06:35:27.028ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5802025-04-30T06:35:27.029ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5812025-04-30T06:35:27.041ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5822025-04-30T06:35:27.042ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5832025-04-30T06:35:27.042ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5842025-04-30T06:35:27.042ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5852025-04-30T06:35:27.043ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5862025-04-30T06:35:27.054ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5872025-04-30T06:35:27.057ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5882025-04-30T06:35:27.057ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5892025-04-30T06:35:27.057ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5902025-04-30T06:35:27.058ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5912025-04-30T06:35:27.077ZTRACtest_metrics (DataLoader): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList
5922025-04-30T06:35:27.081ZTRACtest_metrics (DataLoader): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
5932025-04-30T06:35:27.082ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5942025-04-30T06:35:27.082ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5952025-04-30T06:35:27.083ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
5962025-04-30T06:35:27.085ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
5972025-04-30T06:35:27.088ZDEBGtest_metrics (DataLoader): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = IpPoolList result = Ok(())
5982025-04-30T06:35:27.088ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
5992025-04-30T06:35:27.088ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
6002025-04-30T06:35:27.088ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6012025-04-30T06:35:27.171ZTRACtest_metrics (DataLoader): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database
6022025-04-30T06:35:27.171ZDEBGtest_metrics (DataLoader): roles roles = RoleSet { roles: {} }
6032025-04-30T06:35:27.172ZDEBGtest_metrics (DataLoader): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000001, .. }) resource = Database result = Ok(())
6042025-04-30T06:35:27.178ZINFOtest_metrics (nexus): populate complete
6052025-04-30T06:35:27.178ZINFOtest_metrics (nexus): activating background tasks
6062025-04-30T06:35:27.182ZDEBGtest_metrics (init_with_steps): Running step start_sled1
6072025-04-30T06:35:27.182ZINFOtest_metrics (omicron_sled_agent::sim::Server): setting up sled agent server sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6082025-04-30T06:35:27.265ZINFOtest_metrics (SledAgent): created simulated sled agent server = b6d65341-167c-41df-9b5c-41cded99c229 sim_mode = Explicit sled_id = b6d65341-167c-41df-9b5c-41cded99c229
6092025-04-30T06:35:27.265ZINFOtest_metrics (omicron_sled_agent::sim::SimulatedUpstairs): registered sled b6d65341-167c-41df-9b5c-41cded99c229 storage
6102025-04-30T06:35:27.266ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 1 reason = Timeout
6112025-04-30T06:35:27.266ZDEBGtest_metrics (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
6122025-04-30T06:35:27.266ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
6132025-04-30T06:35:27.266ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
6142025-04-30T06:35:27.267ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
6152025-04-30T06:35:27.267ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 1 reason = Timeout
6162025-04-30T06:35:27.268ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
6172025-04-30T06:35:27.268ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 1 reason = Timeout
6182025-04-30T06:35:27.268ZWARNtest_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config nor servers
6192025-04-30T06:35:27.270ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 1.648965ms iteration = 1 status = {"error":"no config nor servers"}
6202025-04-30T06:35:27.270ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 1 reason = Timeout
6212025-04-30T06:35:27.270ZDEBGtest_metrics (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
6222025-04-30T06:35:27.270ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
6232025-04-30T06:35:27.270ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
6242025-04-30T06:35:27.288ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
6252025-04-30T06:35:27.288ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 1 reason = Timeout
6262025-04-30T06:35:27.288ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
6272025-04-30T06:35:27.288ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 1 reason = Timeout
6282025-04-30T06:35:27.288ZWARNtest_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config nor servers
6292025-04-30T06:35:27.288ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 61.382\u{b5}s iteration = 1 status = {"error":"no config nor servers"}
6302025-04-30T06:35:27.288ZDEBGtest_metrics (BackgroundTasks): activating background_task = metrics_producer_gc iteration = 1 reason = Timeout
6312025-04-30T06:35:27.289ZINFOtest_metrics (BackgroundTasks): Metric producer GC running background_task = metrics_producer_gc expiration = 2025-04-30 06:25:27.288920065 UTC
6322025-04-30T06:35:27.295ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database
6332025-04-30T06:35:27.295ZDEBGtest_metrics (BackgroundTasks): roles background_task = metrics_producer_gc roles = RoleSet { roles: {} }
6342025-04-30T06:35:27.296ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = metrics_producer_gc resource = Database result = Ok(())
6352025-04-30T06:35:27.296ZDEBGtest_metrics (BackgroundTasks): activating background_task = external_endpoints iteration = 1 reason = Timeout
6362025-04-30T06:35:27.297ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
6372025-04-30T06:35:27.297ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6382025-04-30T06:35:27.297ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
6392025-04-30T06:35:27.297ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
6402025-04-30T06:35:27.297ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
6412025-04-30T06:35:27.298ZDEBGtest_metrics (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 1 reason = Timeout
6422025-04-30T06:35:27.298ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
6432025-04-30T06:35:27.298ZDEBGtest_metrics (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
6442025-04-30T06:35:27.299ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
6452025-04-30T06:35:27.299ZDEBGtest_metrics (BackgroundTasks): activating background_task = bfd_manager iteration = 1 reason = Timeout
6462025-04-30T06:35:27.300ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
6472025-04-30T06:35:27.300ZDEBGtest_metrics (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
6482025-04-30T06:35:27.300ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
6492025-04-30T06:35:27.300ZDEBGtest_metrics (BackgroundTasks): activating background_task = phantom_disks iteration = 1 reason = Timeout
6502025-04-30T06:35:27.300ZDEBGtest_metrics (BackgroundTasks): activating background_task = blueprint_loader iteration = 1 reason = Timeout
6512025-04-30T06:35:27.301ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig
6522025-04-30T06:35:27.301ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = blueprint_loader resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6532025-04-30T06:35:27.301ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
6542025-04-30T06:35:27.301ZDEBGtest_metrics (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
6552025-04-30T06:35:27.301ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
6562025-04-30T06:35:27.302ZDEBGtest_metrics (BackgroundTasks): activating background_task = blueprint_executor iteration = 1 reason = Timeout
6572025-04-30T06:35:27.302ZWARNtest_metrics (BackgroundTasks): Blueprint execution: skipped background_task = blueprint_executor reason = no blueprint
6582025-04-30T06:35:27.302ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = blueprint_executor elapsed = 219.818\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6592025-04-30T06:35:27.302ZDEBGtest_metrics (BackgroundTasks): activating background_task = crdb_node_id_collector iteration = 1 reason = Timeout
6602025-04-30T06:35:27.302ZWARNtest_metrics (BackgroundTasks): Blueprint execution: skipped background_task = crdb_node_id_collector reason = no blueprint
6612025-04-30T06:35:27.302ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = crdb_node_id_collector elapsed = 164.849\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6622025-04-30T06:35:27.302ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 1 reason = Timeout
6632025-04-30T06:35:27.302ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
6642025-04-30T06:35:27.303ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6652025-04-30T06:35:27.303ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
6662025-04-30T06:35:27.303ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
6672025-04-30T06:35:27.303ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
6682025-04-30T06:35:27.304ZDEBGtest_metrics (BackgroundTasks): activating background_task = support_bundle_collector iteration = 1 reason = Timeout
6692025-04-30T06:35:27.304ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
6702025-04-30T06:35:27.304ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6712025-04-30T06:35:27.304ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
6722025-04-30T06:35:27.304ZDEBGtest_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
6732025-04-30T06:35:27.304ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
6742025-04-30T06:35:27.305ZDEBGtest_metrics (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 1 reason = Timeout
6752025-04-30T06:35:27.305ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 96.632\u{b5}s iteration = 1 status = {"error":"task disabled"}
6762025-04-30T06:35:27.305ZDEBGtest_metrics (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 1 reason = Timeout
6772025-04-30T06:35:27.305ZWARNtest_metrics (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
6782025-04-30T06:35:27.305ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 173.977\u{b5}s iteration = 1 status = {"error":"no blueprint"}
6792025-04-30T06:35:27.305ZDEBGtest_metrics (BackgroundTasks): activating background_task = decommissioned_disk_cleaner iteration = 1 reason = Timeout
6802025-04-30T06:35:27.305ZINFOtest_metrics (BackgroundTasks): Decommissioned Disk Cleaner disabled background_task = decommissioned_disk_cleaner
6812025-04-30T06:35:27.305ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = decommissioned_disk_cleaner elapsed = 235.229\u{b5}s iteration = 1 status = {"deleted":0,"error":null,"error_count":0,"found":0,"not_ready_to_be_deleted":0}
6822025-04-30T06:35:27.306ZDEBGtest_metrics (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 1 reason = Timeout
6832025-04-30T06:35:27.306ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
6842025-04-30T06:35:27.306ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6852025-04-30T06:35:27.306ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
6862025-04-30T06:35:27.306ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
6872025-04-30T06:35:27.306ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
6882025-04-30T06:35:27.307ZDEBGtest_metrics (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 1 reason = Timeout
6892025-04-30T06:35:27.307ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
6902025-04-30T06:35:27.307ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
6912025-04-30T06:35:27.307ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
6922025-04-30T06:35:27.307ZDEBGtest_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
6932025-04-30T06:35:27.307ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
6942025-04-30T06:35:27.308ZDEBGtest_metrics (BackgroundTasks): activating background_task = v2p_manager iteration = 1 reason = Timeout
6952025-04-30T06:35:27.310ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
6962025-04-30T06:35:27.310ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
6972025-04-30T06:35:27.311ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
6982025-04-30T06:35:27.313ZDEBGtest_metrics (BackgroundTasks): activating background_task = region_replacement iteration = 1 reason = Timeout
6992025-04-30T06:35:27.325ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
7002025-04-30T06:35:27.325ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
7012025-04-30T06:35:27.332ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
7022025-04-30T06:35:27.332ZDEBGtest_metrics (BackgroundTasks): activating background_task = region_replacement_driver iteration = 1 reason = Timeout
7032025-04-30T06:35:27.332ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
7042025-04-30T06:35:27.332ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
7052025-04-30T06:35:27.333ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
7062025-04-30T06:35:27.333ZDEBGtest_metrics (BackgroundTasks): activating background_task = instance_watcher iteration = 1 reason = Timeout
7072025-04-30T06:35:27.333ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
7082025-04-30T06:35:27.333ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7092025-04-30T06:35:27.333ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
7102025-04-30T06:35:27.333ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
7112025-04-30T06:35:27.334ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
7122025-04-30T06:35:27.334ZDEBGtest_metrics (BackgroundTasks): activating background_task = instance_updater iteration = 1 reason = Timeout
7132025-04-30T06:35:27.334ZINFOtest_metrics (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
7142025-04-30T06:35:27.338ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = instance_updater elapsed = 4.21633ms iteration = 1 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
7152025-04-30T06:35:27.338ZDEBGtest_metrics (BackgroundTasks): activating background_task = instance_reincarnation iteration = 1 reason = Timeout
7162025-04-30T06:35:27.339ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
7172025-04-30T06:35:27.339ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
7182025-04-30T06:35:27.340ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
7192025-04-30T06:35:27.340ZDEBGtest_metrics (BackgroundTasks): activating background_task = service_firewall_rule_propagation iteration = 1 reason = Timeout
7202025-04-30T06:35:27.340ZDEBGtest_metrics (service-firewall-rule-progator): starting background task for service firewall rule propagation background_task = service_firewall_rule_propagation
7212025-04-30T06:35:27.340ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
7222025-04-30T06:35:27.340ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
7232025-04-30T06:35:27.340ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
7242025-04-30T06:35:27.341ZDEBGtest_metrics (BackgroundTasks): activating background_task = vpc_route_manager iteration = 1 reason = Timeout
7252025-04-30T06:35:27.341ZINFOtest_metrics (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
7262025-04-30T06:35:27.341ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
7272025-04-30T06:35:27.341ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7282025-04-30T06:35:27.341ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
7292025-04-30T06:35:27.341ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
7302025-04-30T06:35:27.342ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
7312025-04-30T06:35:27.342ZDEBGtest_metrics (BackgroundTasks): activating background_task = abandoned_vmm_reaper iteration = 1 reason = Timeout
7322025-04-30T06:35:27.342ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database
7332025-04-30T06:35:27.342ZDEBGtest_metrics (BackgroundTasks): roles background_task = abandoned_vmm_reaper roles = RoleSet { roles: {} }
7342025-04-30T06:35:27.343ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = abandoned_vmm_reaper resource = Database result = Ok(())
7352025-04-30T06:35:27.343ZDEBGtest_metrics (BackgroundTasks): activating background_task = saga_recovery iteration = 1 reason = Timeout
7362025-04-30T06:35:27.343ZDEBGtest_metrics (SagaRecoverer): listing candidate sagas for recovery
7372025-04-30T06:35:27.343ZTRACtest_metrics (SagaRecoverer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database
7382025-04-30T06:35:27.343ZDEBGtest_metrics (SagaRecoverer): roles roles = RoleSet { roles: {} }
7392025-04-30T06:35:27.343ZDEBGtest_metrics (SagaRecoverer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000003a8a, .. }) resource = Database result = Ok(())
7402025-04-30T06:35:27.344ZDEBGtest_metrics (BackgroundTasks): activating background_task = lookup_region_port iteration = 1 reason = Timeout
7412025-04-30T06:35:27.344ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database
7422025-04-30T06:35:27.344ZDEBGtest_metrics (BackgroundTasks): roles background_task = lookup_region_port roles = RoleSet { roles: {} }
7432025-04-30T06:35:27.344ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = lookup_region_port resource = Database result = Ok(())
7442025-04-30T06:35:27.344ZDEBGtest_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_start iteration = 1 reason = Timeout
7452025-04-30T06:35:27.344ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
7462025-04-30T06:35:27.344ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
7472025-04-30T06:35:27.345ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
7482025-04-30T06:35:27.345ZDEBGtest_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_garbage_collection iteration = 1 reason = Timeout
7492025-04-30T06:35:27.345ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database
7502025-04-30T06:35:27.345ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_garbage_collection roles = RoleSet { roles: {} }
7512025-04-30T06:35:27.351ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_garbage_collection resource = Database result = Ok(())
7522025-04-30T06:35:27.351ZDEBGtest_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_step iteration = 1 reason = Timeout
7532025-04-30T06:35:27.351ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
7542025-04-30T06:35:27.351ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
7552025-04-30T06:35:27.352ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
7562025-04-30T06:35:27.352ZDEBGtest_metrics (BackgroundTasks): activating background_task = region_snapshot_replacement_finish iteration = 1 reason = Timeout
7572025-04-30T06:35:27.352ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database
7582025-04-30T06:35:27.352ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_finish roles = RoleSet { roles: {} }
7592025-04-30T06:35:27.352ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_finish resource = Database result = Ok(())
7602025-04-30T06:35:27.352ZDEBGtest_metrics (BackgroundTasks): activating background_task = tuf_artifact_replication iteration = 1 reason = Timeout
7612025-04-30T06:35:27.353ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
7622025-04-30T06:35:27.353ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7632025-04-30T06:35:27.353ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
7642025-04-30T06:35:27.353ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
7652025-04-30T06:35:27.353ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
7662025-04-30T06:35:27.353ZDEBGtest_metrics (BackgroundTasks): activating background_task = read_only_region_replacement_start iteration = 1 reason = Timeout
7672025-04-30T06:35:27.353ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database
7682025-04-30T06:35:27.353ZDEBGtest_metrics (BackgroundTasks): roles background_task = read_only_region_replacement_start roles = RoleSet { roles: {} }
7692025-04-30T06:35:27.354ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = read_only_region_replacement_start resource = Database result = Ok(())
7702025-04-30T06:35:27.354ZDEBGtest_metrics (BackgroundTasks): activating background_task = webhook_dispatcher iteration = 1 reason = Timeout
7712025-04-30T06:35:27.354ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
7722025-04-30T06:35:27.354ZDEBGtest_metrics (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
7732025-04-30T06:35:27.354ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
7742025-04-30T06:35:27.354ZDEBGtest_metrics (BackgroundTasks): activating background_task = webhook_deliverator iteration = 1 reason = Timeout
7752025-04-30T06:35:27.355ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database
7762025-04-30T06:35:27.355ZDEBGtest_metrics (BackgroundTasks): roles background_task = webhook_deliverator roles = RoleSet { roles: {} }
7772025-04-30T06:35:27.355ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_deliverator resource = Database result = Ok(())
7782025-04-30T06:35:27.355ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
7792025-04-30T06:35:27.356ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
7802025-04-30T06:35:27.356ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
7812025-04-30T06:35:27.356ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
7822025-04-30T06:35:27.356ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
7832025-04-30T06:35:27.362ZINFOtest_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpC8Yjgy/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7842025-04-30T06:35:27.362ZDEBGtest_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpC8Yjgy/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7852025-04-30T06:35:27.362ZINFOtest_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp9B6Sui/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7862025-04-30T06:35:27.362ZDEBGtest_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp9B6Sui/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7872025-04-30T06:35:27.412ZDEBGtest_metrics (dropshot (Repo Depot)): registered endpoint local_addr = [::1]:54162 method = GET path = /artifact/sha256/{sha256} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7882025-04-30T06:35:27.412ZINFOtest_metrics (dropshot (Repo Depot)): listening local_addr = [::1]:54162 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7892025-04-30T06:35:27.413ZDEBGtest_metrics (dropshot (Repo Depot)): successfully registered DTrace USDT probes local_addr = [::1]:54162 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
7902025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /artifacts sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7912025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /artifacts/{sha256} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7922025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /artifacts/{sha256}/copy-from-depot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7932025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7942025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /artifacts-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7952025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /boot-disk/{boot_disk}/os/write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7962025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /boot-disk/{boot_disk}/os/write/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7972025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = DELETE path = /boot-disk/{boot_disk}/os/write/status/{update_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7982025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /bootstore/status sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
7992025-04-30T06:35:27.419ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /datasets sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8002025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /disks/{disk_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8012025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /disks/{disk_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8022025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /eip-gateways sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8032025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /inventory sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8042025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8052025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /network-bootstore-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8062025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /omicron-config sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8072025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /omicron-physical-disks sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8082025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /sled-identifiers sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8092025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /sled-role sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8102025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /sleds sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8112025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/dladm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8122025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/ipadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8132025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/logs/download/{zone} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8142025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/logs/zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8152025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/nvmeadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8162025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/pargs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8172025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/pfiles-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8182025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/pstack-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8192025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/zfs-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8202025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/zoneadm-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8212025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support/zpool-info sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8222025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support-bundles/{zpool_id}/{dataset_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8232025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = DELETE path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8242025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8252025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8262025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8272025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8282025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/download/{file} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8292025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8302025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = HEAD path = /support-bundles/{zpool_id}/{dataset_id}/{support_bundle_id}/index sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8312025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /switch-ports sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8322025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /timesync sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8332025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = DELETE path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8342025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8352025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /v2p sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8362025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = DELETE path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8372025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /vmms/{propolis_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8382025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /vmms/{propolis_id}/disks/{disk_id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8392025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = DELETE path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8402025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /vmms/{propolis_id}/external-ip sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8412025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /vmms/{propolis_id}/poke sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8422025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /vmms/{propolis_id}/poke-single-step sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8432025-04-30T06:35:27.420ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /vmms/{propolis_id}/sim-migration-source sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8442025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8452025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /vmms/{propolis_id}/state sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8462025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /vpc/{vpc_id}/firewall/rules sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8472025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8482025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /vpc-routes sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8492025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zones sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8502025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = POST path = /zones/bundle-cleanup sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8512025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8522025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = PUT path = /zones/bundle-cleanup/context sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8532025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zones/bundle-cleanup/utilization sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8542025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zones/bundles sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8552025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zones/bundles/{zone_name} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8562025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = DELETE path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8572025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zones/bundles/{zone_name}/{bundle_id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8582025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:56237 method = GET path = /zpools sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
8592025-04-30T06:35:27.421ZINFOtest_metrics (dropshot): listening local_addr = [::1]:56237 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8602025-04-30T06:35:27.421ZDEBGtest_metrics (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:56237 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8612025-04-30T06:35:27.423ZDEBGtest_metrics (omicron_sled_agent::sim::Server): contacting server nexus sled_id = b6d65341-167c-41df-9b5c-41cded99c229
8622025-04-30T06:35:27.423ZDEBGtest_metrics (NexusClient): client request body = Some(Body) method = POST sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = http://127.0.0.1:49095/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8632025-04-30T06:35:27.424ZTRACtest_metrics (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: 49 21 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 I!..........._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:58501 req_id = f5589878-176a-426c-b5a3-4749d1441009
8642025-04-30T06:35:27.424ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18721,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58501 req_id = f5589878-176a-426c-b5a3-4749d1441009
8652025-04-30T06:35:27.425ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 18721, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_nameservice._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 12, 95, 110, 97, 109, 101, 115, 101, 114, 118, 105, 99, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_nameservice._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58501 req_id = f5589878-176a-426c-b5a3-4749d1441009
8662025-04-30T06:35:27.425ZTRACtest_metrics (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: bb 1e 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 ............._ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55363 req_id = 3d35fa0b-0bc5-46ad-9df4-66ad05c23e45
8672025-04-30T06:35:27.425ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47902,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55363 req_id = 3d35fa0b-0bc5-46ad-9df4-66ad05c23e45
8682025-04-30T06:35:27.425ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 47902, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_external-dns._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 13, 95, 101, 120, 116, 101, 114, 110, 97, 108, 45, 100, 110, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_external-dns._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55363 req_id = 3d35fa0b-0bc5-46ad-9df4-66ad05c23e45
8692025-04-30T06:35:27.426ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
8702025-04-30T06:35:27.426ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
8712025-04-30T06:35:27.426ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
8722025-04-30T06:35:27.426ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
8732025-04-30T06:35:27.427ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
8742025-04-30T06:35:27.428ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:59657
8752025-04-30T06:35:27.428ZWARNtest_metrics (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_internal dns_group = internal error = no record found for Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8762025-04-30T06:35:27.428ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 161.016966ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_nameservice._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8772025-04-30T06:35:27.429ZWARNtest_metrics (BackgroundTasks): failed to lookup DNS servers background_task = dns_servers_external dns_group = external error = no record found for Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8782025-04-30T06:35:27.429ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 140.59644ms iteration = 1 status = {"error":"failed to read list of DNS servers: no record found for Query { name: Name(\\"_external-dns._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8792025-04-30T06:35:27.431ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:59657 req_id = 8f2f3787-3b55-4424-821b-5090fd1e4159 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
8802025-04-30T06:35:27.434ZINFOtest_metrics (nexus): registered sled agent sled_uuid = b6d65341-167c-41df-9b5c-41cded99c229
8812025-04-30T06:35:27.458ZINFOtest_metrics (BackgroundTasks): Metric producer GC complete (no errors) background_task = metrics_producer_gc expiration = 2025-04-30 06:25:27.288920065 UTC pruned = {}
8822025-04-30T06:35:27.458ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = metrics_producer_gc elapsed = 169.973882ms iteration = 1 status = {"expiration":"2025-04-30T06:25:27.288920065Z","pruned":[]}
8832025-04-30T06:35:27.463ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8842025-04-30T06:35:27.469ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
8852025-04-30T06:35:27.470ZDEBGtest_metrics (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
8862025-04-30T06:35:27.485ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = BlueprintConfig result = Ok(())
8872025-04-30T06:35:27.485ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database
8882025-04-30T06:35:27.485ZDEBGtest_metrics (BackgroundTasks): roles background_task = blueprint_loader roles = RoleSet { roles: {} }
8892025-04-30T06:35:27.485ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = blueprint_loader resource = Database result = Ok(())
8902025-04-30T06:35:27.487ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 186.145132ms iteration = 1 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
8912025-04-30T06:35:27.489ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 0f 0c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:57128 req_id = 877802ba-cfcc-42cb-bab3-b5eb6202ddec
8922025-04-30T06:35:27.489ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3852,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57128 req_id = 877802ba-cfcc-42cb-bab3-b5eb6202ddec
8932025-04-30T06:35:27.490ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3852, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57128 req_id = 877802ba-cfcc-42cb-bab3-b5eb6202ddec
8942025-04-30T06:35:27.491ZERROtest_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8952025-04-30T06:35:27.494ZERROtest_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = nat_v4_garbage_collector error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
8962025-04-30T06:35:27.494ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = nat_v4_garbage_collector elapsed = 196.006009ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
8972025-04-30T06:35:27.499ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 6d c7 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 m............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52804 req_id = 5dc24804-d859-49c6-a703-2877744530b7
8982025-04-30T06:35:27.499ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28103,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52804 req_id = 5dc24804-d859-49c6-a703-2877744530b7
8992025-04-30T06:35:27.500ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28103, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_dendrite._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 9, 95, 100, 101, 110, 100, 114, 105, 116, 101, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_dendrite._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52804 req_id = 5dc24804-d859-49c6-a703-2877744530b7
9002025-04-30T06:35:27.516ZERROtest_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9012025-04-30T06:35:27.532ZERROtest_metrics (BackgroundTasks): failed to resolve addresses for Dendrite services background_task = bfd_manager error = no record found for Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9022025-04-30T06:35:27.532ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = bfd_manager elapsed = 232.62471ms iteration = 1 status = {"error":"failed to resolve addresses for Dendrite services: no record found for Query { name: Name(\\"_dendrite._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9032025-04-30T06:35:27.532ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9042025-04-30T06:35:27.542ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
9052025-04-30T06:35:27.564ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9062025-04-30T06:35:27.565ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9072025-04-30T06:35:27.566ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9082025-04-30T06:35:27.567ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9092025-04-30T06:35:27.578ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
9102025-04-30T06:35:27.578ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
9112025-04-30T06:35:27.578ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
9122025-04-30T06:35:27.578ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
9132025-04-30T06:35:27.638ZDEBGtest_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9142025-04-30T06:35:27.642ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
9152025-04-30T06:35:27.642ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
9162025-04-30T06:35:27.642ZDEBGtest_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
9172025-04-30T06:35:27.642ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
9182025-04-30T06:35:27.642ZDEBGtest_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9192025-04-30T06:35:27.646ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
9202025-04-30T06:35:27.646ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
9212025-04-30T06:35:27.646ZDEBGtest_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
9222025-04-30T06:35:27.647ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
9232025-04-30T06:35:27.647ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9242025-04-30T06:35:27.653ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
9252025-04-30T06:35:27.653ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
9262025-04-30T06:35:27.653ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
9272025-04-30T06:35:27.653ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
9282025-04-30T06:35:27.692ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database
9292025-04-30T06:35:27.693ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_replacement roles = RoleSet { roles: {} }
9302025-04-30T06:35:27.693ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement resource = Database result = Ok(())
9312025-04-30T06:35:27.716ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
9322025-04-30T06:35:27.716ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
9332025-04-30T06:35:27.717ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
9342025-04-30T06:35:27.718ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9352025-04-30T06:35:27.722ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
9362025-04-30T06:35:27.722ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
9372025-04-30T06:35:27.722ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9382025-04-30T06:35:27.722ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
9392025-04-30T06:35:27.722ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
9402025-04-30T06:35:27.723ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
9412025-04-30T06:35:27.723ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9422025-04-30T06:35:27.727ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
9432025-04-30T06:35:27.727ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
9442025-04-30T06:35:27.727ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
9452025-04-30T06:35:27.727ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
9462025-04-30T06:35:27.803ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database
9472025-04-30T06:35:27.823ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_replacement_driver roles = RoleSet { roles: {} }
9482025-04-30T06:35:27.823ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_replacement_driver resource = Database result = Ok(())
9492025-04-30T06:35:27.852ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = lookup_region_port elapsed = 508.336217ms iteration = 1 status = {"errors":[],"found_port_ok":[]}
9502025-04-30T06:35:27.957ZDEBGtest_metrics (BackgroundTasks): Found abandoned VMMs background_task = abandoned_vmm_reaper count = 0 total = 0
9512025-04-30T06:35:27.958ZINFOtest_metrics (BackgroundTasks): Abandoned VMMs reaped background_task = abandoned_vmm_reaper sled_reservations_deleted = 0 vmms_already_deleted = 0 vmms_deleted = 0 vmms_found = 0
9522025-04-30T06:35:27.958ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = abandoned_vmm_reaper elapsed = 615.403943ms iteration = 1 status = {"errors":[],"sled_reservations_deleted":0,"vmms_already_deleted":0,"vmms_deleted":0,"vmms_found":0}
9532025-04-30T06:35:27.974ZTRACtest_metrics (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = instance failed total_found = 0
9542025-04-30T06:35:27.974ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database
9552025-04-30T06:35:27.974ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_reincarnation roles = RoleSet { roles: {} }
9562025-04-30T06:35:27.975ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_reincarnation resource = Database result = Ok(())
9572025-04-30T06:35:27.975ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
9582025-04-30T06:35:27.975ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
9592025-04-30T06:35:27.976ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
9602025-04-30T06:35:27.986ZINFOtest_metrics (SagaRecoverer): listed in-progress sagas count = 0
9612025-04-30T06:35:27.986ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = saga_recovery elapsed = 643.247218ms iteration = 1 status = {"last_pass":{"Success":{"nfailed":0,"nfound":0,"nrecovered":0,"nremoved":0,"nskipped":0}},"ntotal_failures":0,"ntotal_finished":0,"ntotal_recovered":0,"ntotal_sec_errors_bad_state":0,"ntotal_sec_errors_missing":0,"ntotal_started":0,"recent_failures":[],"recent_recoveries":[]}
9622025-04-30T06:35:27.992ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9632025-04-30T06:35:27.996ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
9642025-04-30T06:35:27.996ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
9652025-04-30T06:35:27.996ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9662025-04-30T06:35:27.996ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
9672025-04-30T06:35:27.996ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
9682025-04-30T06:35:27.996ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
9692025-04-30T06:35:28.040ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9702025-04-30T06:35:28.047ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
9712025-04-30T06:35:28.140ZWARNtest_metrics (BackgroundTasks): failed to read target blueprint background_task = blueprint_loader error = Internal Error: no target blueprint set
9722025-04-30T06:35:28.141ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = blueprint_loader elapsed = 840.969598ms iteration = 1 status = {"error":"failed to read target blueprint: Internal Error: no target blueprint set"}
9732025-04-30T06:35:28.142ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = webhook_deliverator elapsed = 787.680673ms iteration = 1 status = {"by_rx":{},"error":null}
9742025-04-30T06:35:28.142ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
9752025-04-30T06:35:28.142ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
9762025-04-30T06:35:28.144ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
9772025-04-30T06:35:28.145ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
9782025-04-30T06:35:28.145ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
9792025-04-30T06:35:28.145ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
9802025-04-30T06:35:28.145ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
9812025-04-30T06:35:28.145ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
9822025-04-30T06:35:28.151ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database
9832025-04-30T06:35:28.151ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_start roles = RoleSet { roles: {} }
9842025-04-30T06:35:28.151ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_start resource = Database result = Ok(())
9852025-04-30T06:35:28.186ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_garbage_collection elapsed = 841.173143ms iteration = 1 status = {"errors":[],"garbage_collect_requested":[]}
9862025-04-30T06:35:28.190ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
9872025-04-30T06:35:28.190ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
9882025-04-30T06:35:28.191ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 9a b0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61382 req_id = 109d765d-90e3-4b82-89dd-1f4d8f3c4915
9892025-04-30T06:35:28.191ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39600,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61382 req_id = 109d765d-90e3-4b82-89dd-1f4d8f3c4915
9902025-04-30T06:35:28.191ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 39600, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_mgs._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 4, 95, 109, 103, 115, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } SERVFAIL: server is not authoritative for name: "_mgs._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61382 req_id = 109d765d-90e3-4b82-89dd-1f4d8f3c4915
9912025-04-30T06:35:28.192ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN }
9922025-04-30T06:35:28.192ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 889.312197ms iteration = 1 status = {"error":"failed to collect inventory: looking up MGS addresses: no record found for Query { name: Name(\\"_mgs._tcp.control-plane.oxide.internal.\\"), query_type: SRV, query_class: IN }"}
9932025-04-30T06:35:28.192ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_finish elapsed = 839.951398ms iteration = 1 status = {"errors":[],"finish_invoked_ok":[]}
9942025-04-30T06:35:28.193ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = switch_port_config_manager elapsed = 886.155407ms iteration = 1 status = {}
9952025-04-30T06:35:28.194ZWARNtest_metrics (BackgroundTasks): inventory collection is None background_task = service_zone_nat_tracker
9962025-04-30T06:35:28.194ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 888.349168ms iteration = 1 status = {"error":"inventory collection is None"}
9972025-04-30T06:35:28.196ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = read_only_region_replacement_start elapsed = 842.912188ms iteration = 1 status = {"errors":[],"requests_created_ok":[]}
9982025-04-30T06:35:28.216ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
9992025-04-30T06:35:28.219ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
10002025-04-30T06:35:28.219ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
10012025-04-30T06:35:28.219ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
10022025-04-30T06:35:28.220ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
10032025-04-30T06:35:28.220ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10042025-04-30T06:35:28.220ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10052025-04-30T06:35:28.221ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10062025-04-30T06:35:28.221ZWARNtest_metrics (BackgroundTasks): failed to read DNS config background_task = dns_config_internal dns_group = internal error = Internal Error: expected exactly one latest version for DNS group Internal, found 0
10072025-04-30T06:35:28.221ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 955.208491ms iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group Internal, found 0"}
10082025-04-30T06:35:28.242ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10092025-04-30T06:35:28.247ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
10102025-04-30T06:35:28.250ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10112025-04-30T06:35:28.254ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
10122025-04-30T06:35:28.254ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
10132025-04-30T06:35:28.254ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
10142025-04-30T06:35:28.254ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
10152025-04-30T06:35:28.255ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = region_replacement elapsed = 942.456604ms iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10162025-04-30T06:35:28.256ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database
10172025-04-30T06:35:28.256ZDEBGtest_metrics (BackgroundTasks): roles background_task = region_snapshot_replacement_step roles = RoleSet { roles: {} }
10182025-04-30T06:35:28.257ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = region_snapshot_replacement_step resource = Database result = Ok(())
10192025-04-30T06:35:28.324ZTRACtest_metrics (BackgroundTasks): reprocessing 0 outdated webhook globs... background_task = webhook_dispatcher
10202025-04-30T06:35:28.324ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database
10212025-04-30T06:35:28.324ZDEBGtest_metrics (BackgroundTasks): roles background_task = webhook_dispatcher roles = RoleSet { roles: {} }
10222025-04-30T06:35:28.325ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = webhook_dispatcher resource = Database result = Ok(())
10232025-04-30T06:35:28.561ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = region_replacement_driver elapsed = 1.229409279s iteration = 1 status = {"drive_invoked_ok":[],"errors":[],"finish_invoked_ok":[]}
10242025-04-30T06:35:28.566ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 2 reason = Signaled
10252025-04-30T06:35:28.572ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
10262025-04-30T06:35:28.573ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10272025-04-30T06:35:28.573ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
10282025-04-30T06:35:28.573ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
10292025-04-30T06:35:28.574ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
10302025-04-30T06:35:28.574ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
10312025-04-30T06:35:28.583ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
10322025-04-30T06:35:28.595ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
10332025-04-30T06:35:28.595ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
10342025-04-30T06:35:28.596ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
10352025-04-30T06:35:28.596ZINFOtest_metrics (dropshot_internal): request completed latency_us = 1165494 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:59657 req_id = 8f2f3787-3b55-4424-821b-5090fd1e4159 response_code = 204 uri = /sled-agents/b6d65341-167c-41df-9b5c-41cded99c229
10362025-04-30T06:35:28.597ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
10372025-04-30T06:35:28.597ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
10382025-04-30T06:35:28.597ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
10392025-04-30T06:35:28.597ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
10402025-04-30T06:35:28.598ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
10412025-04-30T06:35:28.598ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
10422025-04-30T06:35:28.598ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10432025-04-30T06:35:28.598ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
10442025-04-30T06:35:28.598ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
10452025-04-30T06:35:28.599ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
10462025-04-30T06:35:28.600ZDEBGtest_metrics (NexusClient): client response result = Ok(Response { url: "http://127.0.0.1:49095/sled-agents/b6d65341-167c-41df-9b5c-41cded99c229", status: 204, headers: {"x-request-id": "8f2f3787-3b55-4424-821b-5090fd1e4159", "date": "Wed, 30 Apr 2025 06:35:28 GMT"} }) sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10472025-04-30T06:35:28.600ZDEBGtest_metrics (init_with_steps): Running step start_crucible_pantry
10482025-04-30T06:35:28.602ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = GET path = /crucible/pantry/0 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10492025-04-30T06:35:28.602ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = GET path = /crucible/pantry/0/job/{id}/is-finished sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10502025-04-30T06:35:28.602ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = GET path = /crucible/pantry/0/job/{id}/ok sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10512025-04-30T06:35:28.602ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = DELETE path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10522025-04-30T06:35:28.602ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = GET path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10532025-04-30T06:35:28.603ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = POST path = /crucible/pantry/0/volume/{id} sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10542025-04-30T06:35:28.603ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = POST path = /crucible/pantry/0/volume/{id}/background sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10552025-04-30T06:35:28.603ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = POST path = /crucible/pantry/0/volume/{id}/bulk-write sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10562025-04-30T06:35:28.603ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = POST path = /crucible/pantry/0/volume/{id}/import-from-url sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10572025-04-30T06:35:28.603ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = POST path = /crucible/pantry/0/volume/{id}/scrub sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10582025-04-30T06:35:28.603ZDEBGtest_metrics (dropshot): registered endpoint kind = pantry local_addr = [::1]:38907 method = POST path = /crucible/pantry/0/volume/{id}/snapshot sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
10592025-04-30T06:35:28.603ZINFOtest_metrics (dropshot): listening kind = pantry local_addr = [::1]:38907 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10602025-04-30T06:35:28.604ZDEBGtest_metrics (dropshot): successfully registered DTrace USDT probes kind = pantry local_addr = [::1]:38907 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10612025-04-30T06:35:28.604ZINFOtest_metrics (omicron_sled_agent::sim::Server): Started Simulated Crucible Pantry address = [::1]:38907 kind = pantry sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10622025-04-30T06:35:28.604ZDEBGtest_metrics (init_with_steps): Running step populate_internal_dns
10632025-04-30T06:35:28.604ZDEBGtest_metrics: Populating Internal DNS
10642025-04-30T06:35:28.669ZINFOtest_metrics: DNS population: DnsConfigParams { generation: Generation( 1, ), time_created: 2025-04-30T06:35:28.668769325Z, zones: [ DnsConfigZone { zone_name: "control-plane.oxide.internal", records: { "30f2c9a7-28dd-44dc-876c-b0a3f1226597.host": [ Aaaa( ::1, ), ], "d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host": [ Aaaa( ::1, ), ], "_nameservice._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36868, target: "441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal", }, ), ], "_nexus._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 49095, target: "588db90d-5640-4be3-9102-2ebf0af0c9a3.host.control-plane.oxide.internal", }, ), ], "_dendrite._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 46964, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_clickhouse-admin-single-server._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 8888, target: "74ca8414-2268-445a-b519-80219b6ee202.host.control-plane.oxide.internal", }, ), ], "8abe35ca-5518-49a6-8363-901ebe0a98a8.host": [ Aaaa( ::1, ), ], "_mgs._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 36378, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], "_cockroach._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 57004, target: "30f2c9a7-28dd-44dc-876c-b0a3f1226597.host.control-plane.oxide.internal", }, ), ], "_crucible-pantry._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 38907, target: "8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal", }, ), ], "_clickhouse._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 49790, target: "74ca8414-2268-445a-b519-80219b6ee202.host.control-plane.oxide.internal", }, ), ], "_clickhouse-native._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "74ca8414-2268-445a-b519-80219b6ee202.host.control-plane.oxide.internal", }, ), ], "_oximeter-reader._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 9000, target: "74ca8414-2268-445a-b519-80219b6ee202.host.control-plane.oxide.internal", }, ), ], "588db90d-5640-4be3-9102-2ebf0af0c9a3.host": [ Aaaa( ::ffff:127.0.0.1, ), ], "441da737-6ecf-40f8-95fb-d6fbdea9e50e.host": [ Aaaa( ::1, ), ], "b6d65341-167c-41df-9b5c-41cded99c229.sled": [ Aaaa( ::1, ), ], "74ca8414-2268-445a-b519-80219b6ee202.host": [ Aaaa( ::1, ), ], "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host": [ Aaaa( ::1, ), ], "_repo-depot._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 54162, target: "b6d65341-167c-41df-9b5c-41cded99c229.sled.control-plane.oxide.internal", }, ), ], "_external-dns._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 43730, target: "d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal", }, ), ], "_mgd._tcp": [ Srv( Srv { prio: 0, weight: 0, port: 39463, target: "dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal", }, ), ], }, }, ], }
10652025-04-30T06:35:28.693ZDEBGtest_metrics: client request body = Some(Body) method = PUT uri = http://[::1]:36868/config
10662025-04-30T06:35:28.696ZWARNtest_metrics (BackgroundTasks): failed to read DNS config background_task = dns_config_external dns_group = external error = Internal Error: expected exactly one latest version for DNS group External, found 0
10672025-04-30T06:35:28.696ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 1.426242836s iteration = 1 status = {"error":"failed to read DNS config: Internal Error: expected exactly one latest version for DNS group External, found 0"}
10682025-04-30T06:35:28.696ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet
10692025-04-30T06:35:28.697ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = support_bundle_collector resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
10702025-04-30T06:35:28.697ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
10712025-04-30T06:35:28.697ZDEBGtest_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
10722025-04-30T06:35:28.698ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
10732025-04-30T06:35:28.700ZINFOtest_metrics (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
10742025-04-30T06:35:28.700ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 1.367630625s iteration = 1 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
10752025-04-30T06:35:28.701ZTRACtest_metrics (BackgroundTasks): webhook dispatching completed successfully background_task = webhook_dispatcher events_dispatched = 0 events_without_receivers = 0
10762025-04-30T06:35:28.701ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = webhook_dispatcher elapsed = 1.347027706s iteration = 1 status = {"dispatched":[],"errors":[],"glob_version":"140.0.0","globs_reprocessed":{},"no_receivers":[]}
10772025-04-30T06:35:28.701ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_start elapsed = 1.356979614s iteration = 1 status = {"errors":[],"requests_completed_ok":[],"requests_created_ok":[],"start_invoked_ok":[]}
10782025-04-30T06:35:28.701ZINFOtest_metrics (http): accepted connection kind = dns local_addr = [::1]:36868 remote_addr = [::1]:54596
10792025-04-30T06:35:28.703ZTRACtest_metrics (http): incoming request kind = dns local_addr = [::1]:36868 method = PUT remote_addr = [::1]:54596 req_id = aaea4d72-3fca-4979-a0fb-3644bf71ea20 uri = /config
10802025-04-30T06:35:28.704ZINFOtest_metrics (store): attempting generation update new_generation = 1 req_id = aaea4d72-3fca-4979-a0fb-3644bf71ea20
10812025-04-30T06:35:28.704ZINFOtest_metrics (store): pruning trees for generations newer than 0
10822025-04-30T06:35:28.704ZDEBGtest_metrics (store): creating tree tree_name = generation_1_zone_control-plane.oxide.internal
10832025-04-30T06:35:28.705ZDEBGtest_metrics (store): updating current config
10842025-04-30T06:35:28.706ZDEBGtest_metrics (store): flushing default tree
10852025-04-30T06:35:28.706ZINFOtest_metrics (store): pruning trees for generations older than 1 keep = 3
10862025-04-30T06:35:28.706ZINFOtest_metrics (store): updated generation new_generation = 1 req_id = aaea4d72-3fca-4979-a0fb-3644bf71ea20
10872025-04-30T06:35:28.706ZINFOtest_metrics (http): request completed kind = dns latency_us = 3088 local_addr = [::1]:36868 method = PUT remote_addr = [::1]:54596 req_id = aaea4d72-3fca-4979-a0fb-3644bf71ea20 response_code = 204 uri = /config
10882025-04-30T06:35:28.707ZDEBGtest_metrics: client response result = Ok(Response { url: "http://[::1]:36868/config", status: 204, headers: {"x-request-id": "aaea4d72-3fca-4979-a0fb-3644bf71ea20", "date": "Wed, 30 Apr 2025 06:35:28 GMT"} })
10892025-04-30T06:35:28.708ZDEBGtest_metrics (init_with_steps): Running step configure_sled_agents
10902025-04-30T06:35:28.782ZDEBGtest_metrics: client request body = Some(Body) method = PUT uri = http://[::1]:56237/omicron-config
10912025-04-30T06:35:28.783ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:33473 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
10922025-04-30T06:35:28.783ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:33473 req_id = e528a96c-0e3d-4c61-a860-66e72081702d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10932025-04-30T06:35:28.786ZINFOtest_metrics (dropshot): request completed latency_us = 2766 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:33473 req_id = e528a96c-0e3d-4c61-a860-66e72081702d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /omicron-config
10942025-04-30T06:35:28.788ZDEBGtest_metrics: client response result = Ok(Response { url: "http://[::1]:56237/omicron-config", status: 200, headers: {"content-type": "application/json", "x-request-id": "e528a96c-0e3d-4c61-a860-66e72081702d", "content-length": "26", "date": "Wed, 30 Apr 2025 06:35:28 GMT"} })
10952025-04-30T06:35:28.789ZDEBGtest_metrics: client request body = Some(Body) method = PUT uri = http://[::1]:56237/network-bootstore-config
10962025-04-30T06:35:28.789ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:33473 req_id = 119bfa7e-b6ed-45b6-a807-dccb71151683 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10972025-04-30T06:35:28.793ZINFOtest_metrics (dropshot): request completed latency_us = 4084 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:33473 req_id = 119bfa7e-b6ed-45b6-a807-dccb71151683 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /network-bootstore-config
10982025-04-30T06:35:28.794ZDEBGtest_metrics: client response result = Ok(Response { url: "http://[::1]:56237/network-bootstore-config", status: 204, headers: {"x-request-id": "119bfa7e-b6ed-45b6-a807-dccb71151683", "date": "Wed, 30 Apr 2025 06:35:28 GMT"} })
10992025-04-30T06:35:28.795ZDEBGtest_metrics (init_with_steps): Running step start_nexus_external
11002025-04-30T06:35:28.795ZDEBGtest_metrics: Starting Nexus (external API)
11012025-04-30T06:35:31.726ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
11022025-04-30T06:35:31.726ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11032025-04-30T06:35:31.727ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11042025-04-30T06:35:31.727ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
11052025-04-30T06:35:31.728ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11062025-04-30T06:35:31.728ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11072025-04-30T06:35:31.728ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11082025-04-30T06:35:31.728ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11092025-04-30T06:35:31.748ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11102025-04-30T06:35:31.752ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11112025-04-30T06:35:31.761ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
11122025-04-30T06:35:31.761ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
11132025-04-30T06:35:31.761ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
11142025-04-30T06:35:31.762ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
11152025-04-30T06:35:31.765ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = region_snapshot_replacement_step elapsed = 4.414415534s iteration = 1 status = {"errors":[],"step_garbage_collect_invoked_ok":[],"step_invoked_ok":[],"step_records_created_ok":[],"step_set_volume_deleted_ok":[]}
11162025-04-30T06:35:31.822ZINFOtest_metrics (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
11172025-04-30T06:35:31.822ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:56237/vpc-routes
11182025-04-30T06:35:31.823ZTRACtest_metrics (BackgroundTasks): no more instances in need of reincarnation background_task = instance_reincarnation reincarnation_reason = start saga failed total_found = 0
11192025-04-30T06:35:31.823ZINFOtest_metrics (BackgroundTasks): instance reincarnation completed successfully background_task = instance_reincarnation instances_changed_state = 0 instances_found = 0 instances_reincarnated = 0
11202025-04-30T06:35:31.823ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = instance_reincarnation elapsed = 4.484666463s iteration = 1 status = {"changed_state":[],"disabled":false,"errors":[],"instances_found":{"Failed":0,"SagaUnwound":0},"instances_reincarnated":[],"restart_errors":[]}
11212025-04-30T06:35:31.870ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11222025-04-30T06:35:31.870ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11232025-04-30T06:35:31.870ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11242025-04-30T06:35:31.870ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11252025-04-30T06:35:31.871ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11262025-04-30T06:35:31.871ZWARNtest_metrics (BackgroundTasks): failed to read Silo/DNS/TLS configuration background_task = external_endpoints error = Internal Error: expected at least one external DNS zone
11272025-04-30T06:35:31.871ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 4.574578999s iteration = 1 status = {"error":"failed to read Silo/DNS/TLS configuration: Internal Error: expected at least one external DNS zone"}
11282025-04-30T06:35:31.872ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11292025-04-30T06:35:31.875ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11302025-04-30T06:35:31.875ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
11312025-04-30T06:35:31.875ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11322025-04-30T06:35:31.875ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11332025-04-30T06:35:31.875ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11342025-04-30T06:35:31.875ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11352025-04-30T06:35:31.876ZDEBGtest_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11362025-04-30T06:35:31.881ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Fleet result = Ok(())
11372025-04-30T06:35:31.882ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database
11382025-04-30T06:35:31.882ZDEBGtest_metrics (BackgroundTasks): roles background_task = support_bundle_collector roles = RoleSet { roles: {} }
11392025-04-30T06:35:31.883ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = support_bundle_collector resource = Database result = Ok(())
11402025-04-30T06:35:31.884ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:64588 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
11412025-04-30T06:35:31.886ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:64588 req_id = 150fa737-9a2d-4fb1-be92-f57f23a394bc sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11422025-04-30T06:35:31.887ZINFOtest_metrics (dropshot): request completed latency_us = 600 local_addr = [::1]:56237 method = GET remote_addr = [::1]:64588 req_id = 150fa737-9a2d-4fb1-be92-f57f23a394bc response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
11432025-04-30T06:35:31.890ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:56237/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "150fa737-9a2d-4fb1-be92-f57f23a394bc", "content-length": "2", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
11442025-04-30T06:35:31.891ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
11452025-04-30T06:35:31.891ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
11462025-04-30T06:35:31.891ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
11472025-04-30T06:35:31.893ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11482025-04-30T06:35:31.893ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11492025-04-30T06:35:31.893ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11502025-04-30T06:35:31.894ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11512025-04-30T06:35:31.894ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11522025-04-30T06:35:31.898ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
11532025-04-30T06:35:31.899ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
11542025-04-30T06:35:31.899ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
11552025-04-30T06:35:31.900ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
11562025-04-30T06:35:31.901ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = []
11572025-04-30T06:35:31.901ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
11582025-04-30T06:35:31.903ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11592025-04-30T06:35:31.908ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11602025-04-30T06:35:31.908ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11612025-04-30T06:35:31.908ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11622025-04-30T06:35:31.913ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11632025-04-30T06:35:31.921ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6c c2 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 l............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51411 req_id = af7effd3-ae9e-40c0-b884-229bdc10756b
11642025-04-30T06:35:31.922ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27842,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51411 req_id = af7effd3-ae9e-40c0-b884-229bdc10756b
11652025-04-30T06:35:31.922ZDEBGtest_metrics (store): query key key = _mgs._tcp
11662025-04-30T06:35:31.923ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11672025-04-30T06:35:31.923ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51411 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = af7effd3-ae9e-40c0-b884-229bdc10756b
11682025-04-30T06:35:31.924ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3796, tv_nsec: 723790749 } })
11692025-04-30T06:35:31.925ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
11702025-04-30T06:35:31.928ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11712025-04-30T06:35:31.936ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
11722025-04-30T06:35:31.937ZTRACtest_metrics (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
11732025-04-30T06:35:31.937ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
11742025-04-30T06:35:31.937ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
11752025-04-30T06:35:31.937ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11762025-04-30T06:35:31.937ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11772025-04-30T06:35:31.937ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11782025-04-30T06:35:31.944ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c5 75 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .u..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51227 req_id = 58a7e892-cf27-42b9-9c7e-be27a1f19f68
11792025-04-30T06:35:31.944ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50549,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51227 req_id = 58a7e892-cf27-42b9-9c7e-be27a1f19f68
11802025-04-30T06:35:31.945ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
11812025-04-30T06:35:31.945ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51227 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 58a7e892-cf27-42b9-9c7e-be27a1f19f68
11822025-04-30T06:35:31.945ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet
11832025-04-30T06:35:31.945ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = tuf_artifact_replication resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
11842025-04-30T06:35:31.945ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11852025-04-30T06:35:31.945ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
11862025-04-30T06:35:31.946ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
11872025-04-30T06:35:31.946ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
11882025-04-30T06:35:31.946ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
11892025-04-30T06:35:31.946ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
11902025-04-30T06:35:31.947ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
11912025-04-30T06:35:31.948ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11922025-04-30T06:35:31.958ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
11932025-04-30T06:35:31.959ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
11942025-04-30T06:35:31.959ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
11952025-04-30T06:35:31.959ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
11962025-04-30T06:35:31.960ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
11972025-04-30T06:35:31.966ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Fleet result = Ok(())
11982025-04-30T06:35:31.966ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database
11992025-04-30T06:35:31.966ZDEBGtest_metrics (BackgroundTasks): roles background_task = tuf_artifact_replication roles = RoleSet { roles: {} }
12002025-04-30T06:35:31.968ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = tuf_artifact_replication resource = Database result = Ok(())
12012025-04-30T06:35:31.968ZINFOtest_metrics (BackgroundTasks): No bundles to collect background_task = support_bundle_collector
12022025-04-30T06:35:31.968ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = support_bundle_collector elapsed = 4.664850639s iteration = 1 status = {"cleanup_err":null,"cleanup_report":{"db_destroying_bundles_removed":0,"db_failing_bundles_updated":0,"sled_bundles_delete_failed":0,"sled_bundles_deleted_not_found":0,"sled_bundles_deleted_ok":0},"collection_err":null,"collection_report":null}
12032025-04-30T06:35:31.969ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12042025-04-30T06:35:31.969ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12052025-04-30T06:35:31.970ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12062025-04-30T06:35:32.083ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
12072025-04-30T06:35:32.089ZINFOtest_metrics (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
12082025-04-30T06:35:32.090ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:56237/eip-gateways
12092025-04-30T06:35:32.090ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12102025-04-30T06:35:32.090ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12112025-04-30T06:35:32.090ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12122025-04-30T06:35:32.091ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12132025-04-30T06:35:32.100ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = Some(Body) method = PUT uri = http://[::1]:56237/artifacts-config
12142025-04-30T06:35:32.102ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12152025-04-30T06:35:32.128ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
12162025-04-30T06:35:32.128ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12172025-04-30T06:35:32.128ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12182025-04-30T06:35:32.129ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12192025-04-30T06:35:32.135ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:64588 req_id = 6dfb7553-215f-431c-8378-5c0da9889af9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12202025-04-30T06:35:32.136ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:53490 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12212025-04-30T06:35:32.143ZINFOtest_metrics (dropshot): request completed latency_us = 7187 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:64588 req_id = 6dfb7553-215f-431c-8378-5c0da9889af9 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
12222025-04-30T06:35:32.143ZINFOtest_metrics (InternalApi): set cockroachdb setting setting = cluster.preserve_downgrade_option value = 22.1
12232025-04-30T06:35:32.144ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12242025-04-30T06:35:32.144ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12252025-04-30T06:35:32.145ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12262025-04-30T06:35:32.146ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:53490 req_id = 062ff009-b727-4ca3-b0a1-6f09cb9233a7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12272025-04-30T06:35:32.146ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 27 2d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c '-..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58219 req_id = b3b3f585-6b3e-40f9-9a0a-7e214c58f45a
12282025-04-30T06:35:32.146ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10029,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58219 req_id = b3b3f585-6b3e-40f9-9a0a-7e214c58f45a
12292025-04-30T06:35:32.146ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
12302025-04-30T06:35:32.146ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10029, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58219 req_id = b3b3f585-6b3e-40f9-9a0a-7e214c58f45a
12312025-04-30T06:35:32.147ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:56237/eip-gateways", status: 204, headers: {"x-request-id": "6dfb7553-215f-431c-8378-5c0da9889af9", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
12322025-04-30T06:35:32.148ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 4.807052096s iteration = 1 status = {}
12332025-04-30T06:35:32.149ZINFOtest_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmpC8Yjgy/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12342025-04-30T06:35:32.149ZDEBGtest_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmpC8Yjgy/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12352025-04-30T06:35:32.149ZINFOtest_metrics (ArtifactStore): No ledger in /var/tmp/omicron_tmp/.tmp9B6Sui/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12362025-04-30T06:35:32.149ZDEBGtest_metrics (ArtifactStore): Failed to read ledger: Not found in storage path = /var/tmp/omicron_tmp/.tmp9B6Sui/artifact-config.json server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12372025-04-30T06:35:32.149ZINFOtest_metrics (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmpC8Yjgy/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12382025-04-30T06:35:32.150ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12392025-04-30T06:35:32.150ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12402025-04-30T06:35:32.151ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12412025-04-30T06:35:32.151ZINFOtest_metrics (ArtifactStore): Writing ledger to /var/tmp/omicron_tmp/.tmp9B6Sui/.artifact-config.json.tmp server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12422025-04-30T06:35:32.152ZINFOtest_metrics (ArtifactStore): Starting delete reconciler generation = 1 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12432025-04-30T06:35:32.152ZINFOtest_metrics (dropshot): request completed latency_us = 5982 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:53490 req_id = 062ff009-b727-4ca3-b0a1-6f09cb9233a7 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts-config
12442025-04-30T06:35:32.153ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
12452025-04-30T06:35:32.153ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
12462025-04-30T06:35:32.153ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
12472025-04-30T06:35:32.156ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:56237/artifacts-config", status: 204, headers: {"x-request-id": "062ff009-b727-4ca3-b0a1-6f09cb9233a7", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
12482025-04-30T06:35:32.156ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication body = None method = GET uri = http://[::1]:56237/artifacts
12492025-04-30T06:35:32.156ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12502025-04-30T06:35:32.156ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12512025-04-30T06:35:32.157ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12522025-04-30T06:35:32.157ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
12532025-04-30T06:35:32.157ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12542025-04-30T06:35:32.157ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12552025-04-30T06:35:32.157ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12562025-04-30T06:35:32.158ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12572025-04-30T06:35:32.159ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:53490 req_id = 17af2337-2e87-438e-a8a5-c78142843991 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12582025-04-30T06:35:32.159ZINFOtest_metrics (dropshot): request completed latency_us = 551 local_addr = [::1]:56237 method = GET remote_addr = [::1]:53490 req_id = 17af2337-2e87-438e-a8a5-c78142843991 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /artifacts
12592025-04-30T06:35:32.186ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = tuf_artifact_replication result = Ok(Response { url: "http://[::1]:56237/artifacts", status: 200, headers: {"content-type": "application/json", "x-request-id": "17af2337-2e87-438e-a8a5-c78142843991", "content-length": "26", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
12602025-04-30T06:35:32.187ZINFOtest_metrics (BackgroundTasks): Successfully got artifact list background_task = tuf_artifact_replication sled = http://[::1]:56237
12612025-04-30T06:35:32.189ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = tuf_artifact_replication elapsed = 4.83666651s iteration = 1 status = {"generation":1,"last_run_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"lifetime_counters":{"copy_err":0,"copy_ok":0,"list_err":0,"list_ok":1,"put_config_err":0,"put_config_ok":1,"put_err":0,"put_ok":0},"local_repos":0,"request_debug_ringbuf":[{"generation":1,"operation":"put_config","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:35:32.156317144Z"},{"operation":"list","target_sled":"b6d65341-167c-41df-9b5c-41cded99c229","time":"2025-04-30T06:35:32.186991912Z"}]}
12622025-04-30T06:35:32.190ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12632025-04-30T06:35:32.190ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12642025-04-30T06:35:32.190ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12652025-04-30T06:35:32.307ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:56237/v2p
12662025-04-30T06:35:32.309ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:60138 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
12672025-04-30T06:35:32.310ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
12682025-04-30T06:35:32.316ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
12692025-04-30T06:35:32.316ZDEBGtest_metrics (InternalApi): updating rack subnet for rack c19a698f-c6f9-4a17-ae30-20d711b8f7dc to Some( V6( Ipv6Network { addr: fd00:1122:3344:100::, prefix: 56, }, ), )
12702025-04-30T06:35:32.317ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12712025-04-30T06:35:32.317ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12722025-04-30T06:35:32.318ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12732025-04-30T06:35:32.319ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:60138 req_id = 6d115b33-ab8c-4076-8a2b-050d7c7d7536 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12742025-04-30T06:35:32.330ZINFOtest_metrics (dropshot): request completed latency_us = 11279 local_addr = [::1]:56237 method = GET remote_addr = [::1]:60138 req_id = 6d115b33-ab8c-4076-8a2b-050d7c7d7536 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
12752025-04-30T06:35:32.330ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
12762025-04-30T06:35:32.330ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
12772025-04-30T06:35:32.330ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12782025-04-30T06:35:32.330ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12792025-04-30T06:35:32.331ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12802025-04-30T06:35:32.335ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:56237/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d115b33-ab8c-4076-8a2b-050d7c7d7536", "content-length": "2", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
12812025-04-30T06:35:32.335ZINFOtest_metrics (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
12822025-04-30T06:35:32.335ZINFOtest_metrics (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
12832025-04-30T06:35:32.335ZINFOtest_metrics (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
12842025-04-30T06:35:32.337ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 5.028432602s iteration = 1 status = {}
12852025-04-30T06:35:32.338ZINFOtest_metrics (InternalApi): Using static configuration for external switchports
12862025-04-30T06:35:32.339ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
12872025-04-30T06:35:32.339ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
12882025-04-30T06:35:32.340ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
12892025-04-30T06:35:32.346ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
12902025-04-30T06:35:32.346ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12912025-04-30T06:35:32.346ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12922025-04-30T06:35:32.347ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12932025-04-30T06:35:32.350ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d44ccc20-b1e2-43e6-bbe4-a4a4b8a80ede", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
12942025-04-30T06:35:32.351ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
12952025-04-30T06:35:32.352ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
12962025-04-30T06:35:32.352ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
12972025-04-30T06:35:32.352ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
12982025-04-30T06:35:32.353ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
12992025-04-30T06:35:32.365ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13002025-04-30T06:35:32.378ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
13012025-04-30T06:35:32.378ZTRACtest_metrics (BackgroundTasks): successful lookup of siloed resource "Vpc" using built-in user background_task = service_firewall_rule_propagation
13022025-04-30T06:35:32.393ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13032025-04-30T06:35:32.393ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13042025-04-30T06:35:32.396ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13052025-04-30T06:35:32.403ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13062025-04-30T06:35:32.403ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13072025-04-30T06:35:32.409ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13082025-04-30T06:35:32.418ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13092025-04-30T06:35:32.418ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13102025-04-30T06:35:32.422ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13112025-04-30T06:35:32.423ZINFOtest_metrics (InternalApi): Recording Rack Network Configuration
13122025-04-30T06:35:32.423ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13132025-04-30T06:35:32.423ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13142025-04-30T06:35:32.424ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13152025-04-30T06:35:32.435ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53e203e0-f0c5-40c4-9528-3a4a3622db83", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
13162025-04-30T06:35:32.439ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
13172025-04-30T06:35:32.439ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13182025-04-30T06:35:32.439ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13192025-04-30T06:35:32.440ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13202025-04-30T06:35:32.449ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13212025-04-30T06:35:32.449ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13222025-04-30T06:35:32.449ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13232025-04-30T06:35:32.449ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13242025-04-30T06:35:32.450ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13252025-04-30T06:35:32.456ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13262025-04-30T06:35:32.456ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13272025-04-30T06:35:32.456ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13282025-04-30T06:35:32.456ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13292025-04-30T06:35:32.470ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13302025-04-30T06:35:32.470ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13312025-04-30T06:35:32.470ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13322025-04-30T06:35:32.471ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13332025-04-30T06:35:32.478ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13342025-04-30T06:35:32.492ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13352025-04-30T06:35:32.500ZTRACtest_metrics (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
13362025-04-30T06:35:32.500ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
13372025-04-30T06:35:32.500ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13382025-04-30T06:35:32.500ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13392025-04-30T06:35:32.500ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13402025-04-30T06:35:32.501ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13412025-04-30T06:35:32.511ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13422025-04-30T06:35:32.511ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13432025-04-30T06:35:32.511ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13442025-04-30T06:35:32.528ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13452025-04-30T06:35:32.535ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13462025-04-30T06:35:32.535ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13472025-04-30T06:35:32.536ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13482025-04-30T06:35:32.536ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13492025-04-30T06:35:32.542ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13502025-04-30T06:35:32.550ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
13512025-04-30T06:35:32.550ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13522025-04-30T06:35:32.550ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13532025-04-30T06:35:32.550ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13542025-04-30T06:35:32.559ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13552025-04-30T06:35:32.560ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13562025-04-30T06:35:32.561ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13572025-04-30T06:35:32.562ZTRACtest_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
13582025-04-30T06:35:32.562ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13592025-04-30T06:35:32.562ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13602025-04-30T06:35:32.562ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13612025-04-30T06:35:32.563ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13622025-04-30T06:35:32.567ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13632025-04-30T06:35:32.568ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13642025-04-30T06:35:32.569ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13652025-04-30T06:35:32.572ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13662025-04-30T06:35:32.572ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13672025-04-30T06:35:32.573ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13682025-04-30T06:35:32.574ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13692025-04-30T06:35:32.577ZDEBGtest_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
13702025-04-30T06:35:32.577ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13712025-04-30T06:35:32.577ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13722025-04-30T06:35:32.578ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13732025-04-30T06:35:32.586ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed7eeff3-4a50-4690-8430-5ce29844037d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
13742025-04-30T06:35:32.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
13752025-04-30T06:35:32.587ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13762025-04-30T06:35:32.587ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13772025-04-30T06:35:32.588ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13782025-04-30T06:35:32.591ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: ec60100a-1711-4b8a-a957-c8cb9a5a4cba, lookup_type: ByName("oxide-service-pool") }
13792025-04-30T06:35:32.592ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13802025-04-30T06:35:32.592ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13812025-04-30T06:35:32.592ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13822025-04-30T06:35:32.593ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13832025-04-30T06:35:32.594ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
13842025-04-30T06:35:32.594ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
13852025-04-30T06:35:32.594ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13862025-04-30T06:35:32.594ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13872025-04-30T06:35:32.595ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13882025-04-30T06:35:32.607ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
13892025-04-30T06:35:32.607ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13902025-04-30T06:35:32.607ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
13912025-04-30T06:35:32.608ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
13922025-04-30T06:35:32.608ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
13932025-04-30T06:35:32.616ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: ec60100a-1711-4b8a-a957-c8cb9a5a4cba, lookup_type: ByName("oxide-service-pool") } result = Ok(())
13942025-04-30T06:35:32.616ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
13952025-04-30T06:35:32.616ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
13962025-04-30T06:35:32.617ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
13972025-04-30T06:35:32.625ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
13982025-04-30T06:35:32.625ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
13992025-04-30T06:35:32.625ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14002025-04-30T06:35:32.626ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14012025-04-30T06:35:32.628ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14022025-04-30T06:35:32.637ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14032025-04-30T06:35:32.637ZTRACtest_metrics (BackgroundTasks): successful lookup of siloed resource "VpcSubnet" using built-in user background_task = service_firewall_rule_propagation
14042025-04-30T06:35:32.637ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
14052025-04-30T06:35:32.637ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
14062025-04-30T06:35:32.637ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14072025-04-30T06:35:32.637ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14082025-04-30T06:35:32.638ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14092025-04-30T06:35:32.638ZTRACtest_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: ec60100a-1711-4b8a-a957-c8cb9a5a4cba, lookup_type: ByName("oxide-service-pool") }
14102025-04-30T06:35:32.638ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14112025-04-30T06:35:32.639ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14122025-04-30T06:35:32.639ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
14132025-04-30T06:35:32.640ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14142025-04-30T06:35:32.644ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
14152025-04-30T06:35:32.644ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14162025-04-30T06:35:32.644ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14172025-04-30T06:35:32.645ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14182025-04-30T06:35:32.647ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b9fb73b-55df-4e14-9c81-813bb7204682", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14192025-04-30T06:35:32.647ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
14202025-04-30T06:35:32.647ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14212025-04-30T06:35:32.652ZDEBGtest_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: ec60100a-1711-4b8a-a957-c8cb9a5a4cba, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14222025-04-30T06:35:32.653ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14232025-04-30T06:35:32.653ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14242025-04-30T06:35:32.653ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14252025-04-30T06:35:32.654ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14262025-04-30T06:35:32.660ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14272025-04-30T06:35:32.671ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
14282025-04-30T06:35:32.672ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14292025-04-30T06:35:32.672ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14302025-04-30T06:35:32.672ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14312025-04-30T06:35:32.676ZTRACtest_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: ec60100a-1711-4b8a-a957-c8cb9a5a4cba, lookup_type: ByName("oxide-service-pool") }
14322025-04-30T06:35:32.676ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14332025-04-30T06:35:32.676ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14342025-04-30T06:35:32.676ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
14352025-04-30T06:35:32.676ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14362025-04-30T06:35:32.690ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14372025-04-30T06:35:32.695ZDEBGtest_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = IpPool { parent: Fleet, key: ec60100a-1711-4b8a-a957-c8cb9a5a4cba, lookup_type: ByName("oxide-service-pool") } result = Ok(())
14382025-04-30T06:35:32.696ZDEBGtest_metrics (service-firewall-rule-progator): resolved names for firewall rules background_task = service_firewall_rule_propagation instance_interfaces = {} subnet_interfaces = {} subnet_networks = {Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })], Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })]} vpc_interfaces = {}
14392025-04-30T06:35:32.696ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet
14402025-04-30T06:35:32.696ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_firewall_rule_propagation resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14412025-04-30T06:35:32.696ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14422025-04-30T06:35:32.696ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14432025-04-30T06:35:32.697ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14442025-04-30T06:35:32.703ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
14452025-04-30T06:35:32.703ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
14462025-04-30T06:35:32.703ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
14472025-04-30T06:35:32.703ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
14482025-04-30T06:35:32.704ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
14492025-04-30T06:35:32.704ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14502025-04-30T06:35:32.708ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Fleet result = Ok(())
14512025-04-30T06:35:32.709ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database
14522025-04-30T06:35:32.709ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_firewall_rule_propagation roles = RoleSet { roles: {} }
14532025-04-30T06:35:32.710ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_firewall_rule_propagation resource = Database result = Ok(())
14542025-04-30T06:35:32.717ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
14552025-04-30T06:35:32.721ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
14562025-04-30T06:35:32.728ZTRACtest_metrics (service-firewall-rule-progator): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:06:08.248500Z, allowed_ips: None } background_task = service_firewall_rule_propagation
14572025-04-30T06:35:32.728ZDEBGtest_metrics (service-firewall-rule-progator): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified. background_task = service_firewall_rule_propagation
14582025-04-30T06:35:32.728ZDEBGtest_metrics (service-firewall-rule-progator): resolved firewall rules for sled agents background_task = service_firewall_rule_propagation sled_agent_rules = []
14592025-04-30T06:35:32.728ZDEBGtest_metrics (service-firewall-rule-progator): resolved 0 rules for sleds background_task = service_firewall_rule_propagation
14602025-04-30T06:35:32.734ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a17a104-18be-40f0-8a0d-4b9f624718c1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14612025-04-30T06:35:32.734ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
14622025-04-30T06:35:32.737ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a9dbfc-ba64-458d-96a7-4babb3f757b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14632025-04-30T06:35:32.737ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
14642025-04-30T06:35:32.763ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d5f6ae9-6b23-42c0-9df7-eea5d50df2dd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14652025-04-30T06:35:32.763ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
14662025-04-30T06:35:32.765ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c97d0850-1be3-4851-817b-862842258eae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14672025-04-30T06:35:32.765ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
14682025-04-30T06:35:32.768ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "74540a98-2c9e-4b84-85bc-89d70a20d0db", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14692025-04-30T06:35:32.768ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
14702025-04-30T06:35:32.770ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a8ec023-294f-461f-9b4f-a536e3481fb0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14712025-04-30T06:35:32.770ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
14722025-04-30T06:35:32.771ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a61dd66-d7ab-4860-a9af-3dc84d03510e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14732025-04-30T06:35:32.771ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
14742025-04-30T06:35:32.773ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bb7ce28-537f-4bee-a517-9d30510eb7fb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14752025-04-30T06:35:32.774ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
14762025-04-30T06:35:32.776ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7b714d6-3fdd-4c11-b781-149e14741489", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14772025-04-30T06:35:32.776ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
14782025-04-30T06:35:32.785ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60700920-836e-469b-ac07-018167c02937", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14792025-04-30T06:35:32.785ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
14802025-04-30T06:35:32.787ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb9ecdbe-7794-43fc-afa8-3184599f70cf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14812025-04-30T06:35:32.787ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
14822025-04-30T06:35:32.789ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e93eb51a-6cb8-4fcb-9219-5348d1ae8ae4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14832025-04-30T06:35:32.789ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
14842025-04-30T06:35:32.791ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3ecdfd9-28b0-404c-b08d-3f5025674a7e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14852025-04-30T06:35:32.791ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
14862025-04-30T06:35:32.794ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99f7fc03-cc1f-4426-94e6-8c3417a33b09", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14872025-04-30T06:35:32.794ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
14882025-04-30T06:35:32.796ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79d51304-af8a-4ccf-8557-28a65f4f3fd8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14892025-04-30T06:35:32.796ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
14902025-04-30T06:35:32.797ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a6dd41e-be1d-441e-aec6-ea611f7beca1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14912025-04-30T06:35:32.797ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
14922025-04-30T06:35:32.798ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c9b3b04-90e0-4042-b0f2-438a67ece758", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14932025-04-30T06:35:32.799ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
14942025-04-30T06:35:32.800ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b593aa7-419f-415e-9ba5-b2bd4e121edc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14952025-04-30T06:35:32.800ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
14962025-04-30T06:35:32.809ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "044e69dd-549a-4406-a494-ac45e71f19fb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
14972025-04-30T06:35:32.810ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
14982025-04-30T06:35:32.821ZDEBGtest_metrics (service-firewall-rule-progator): resolved sleds for vpc oxide-services background_task = service_firewall_rule_propagation vpc_to_sled = []
14992025-04-30T06:35:32.821ZDEBGtest_metrics (service-firewall-rule-progator): sending firewall rules to sled agents background_task = service_firewall_rule_propagation
15002025-04-30T06:35:32.821ZINFOtest_metrics (service-firewall-rule-progator): updated firewall rules on 0 sleds background_task = service_firewall_rule_propagation
15012025-04-30T06:35:32.821ZDEBGtest_metrics (service-firewall-rule-progator): successfully propagated service firewall rules background_task = service_firewall_rule_propagation elapsed = 5.480937127s
15022025-04-30T06:35:32.821ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = service_firewall_rule_propagation elapsed = 5.480995393s iteration = 1 status = {}
15032025-04-30T06:35:32.826ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "23ac3531-06cd-4d23-95ed-b4d3d8ffe10f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15042025-04-30T06:35:32.827ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
15052025-04-30T06:35:32.832ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9be2c5bb-87be-43cf-9f02-d03a1aab67fe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15062025-04-30T06:35:32.833ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
15072025-04-30T06:35:32.835ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23067619-fdc6-489f-8ce0-adc8c2961b3f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15082025-04-30T06:35:32.836ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
15092025-04-30T06:35:32.838ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cde3eb3c-9dd1-41d9-bbf1-01c848c140b1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15102025-04-30T06:35:32.841ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
15112025-04-30T06:35:32.843ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7746fd0b-914d-4e19-9bbd-3d5dd5f1f3c9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15122025-04-30T06:35:32.844ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
15132025-04-30T06:35:32.846ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cea2ef8-b951-4047-8185-dad4899c6673", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15142025-04-30T06:35:32.846ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
15152025-04-30T06:35:32.848ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb915cb-face-497b-a301-7cf8787c174e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15162025-04-30T06:35:32.848ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
15172025-04-30T06:35:32.849ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48737e86-1d89-4854-93cb-dd561abb6bee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15182025-04-30T06:35:32.849ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
15192025-04-30T06:35:32.850ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "349224b3-c204-408d-a215-5c847be9ace5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15202025-04-30T06:35:32.850ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
15212025-04-30T06:35:32.851ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08aa2c99-ed14-42cf-bf75-9407b5cc7496", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15222025-04-30T06:35:32.851ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
15232025-04-30T06:35:32.852ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39a82c3e-6290-47ba-8b15-53b344221d55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15242025-04-30T06:35:32.852ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
15252025-04-30T06:35:32.854ZINFOtest_metrics (InternalApi): inserted blueprint blueprint_id = cd5408f6-7b41-4118-a04d-9c302883693a
15262025-04-30T06:35:32.854ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig
15272025-04-30T06:35:32.854ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15282025-04-30T06:35:32.854ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
15292025-04-30T06:35:32.854ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
15302025-04-30T06:35:32.855ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
15312025-04-30T06:35:32.855ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0ffc214-605c-4272-b711-35b102f9da99", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15322025-04-30T06:35:32.855ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
15332025-04-30T06:35:32.858ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4246120-e56b-4b43-974d-62997a4a0789", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15342025-04-30T06:35:32.858ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
15352025-04-30T06:35:32.859ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15362025-04-30T06:35:32.863ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = BlueprintConfig result = Ok(())
15372025-04-30T06:35:32.866ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3e68291-68ac-4b1d-a973-cbff6ed789ce", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15382025-04-30T06:35:32.866ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
15392025-04-30T06:35:32.869ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5cae53d-31cb-47e8-8b66-7bc77d741f9c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15402025-04-30T06:35:32.869ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
15412025-04-30T06:35:32.870ZINFOtest_metrics (InternalApi): No networking records needed for cockroach_db service
15422025-04-30T06:35:32.870ZINFOtest_metrics (InternalApi): No networking records needed for internal_dns service
15432025-04-30T06:35:32.872ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dfc34a8-9012-4668-a5d2-4c0cf64b7209", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15442025-04-30T06:35:32.873ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
15452025-04-30T06:35:32.875ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c078881d-a0d6-4f0a-ab18-ddae5c69e873", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15462025-04-30T06:35:32.875ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
15472025-04-30T06:35:32.877ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "085266d5-9398-4984-b2b4-15c52f6ebcd5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15482025-04-30T06:35:32.877ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
15492025-04-30T06:35:32.878ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2708c0e0-4609-443b-ab8c-23ae5e1db97a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15502025-04-30T06:35:32.878ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
15512025-04-30T06:35:32.880ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8731a44-003f-4ca8-b3fa-08fd5a20cc3e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15522025-04-30T06:35:32.881ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
15532025-04-30T06:35:32.882ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dda9d01d-7467-4214-8ea7-920c3f49954d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:31 GMT"} })
15542025-04-30T06:35:32.883ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
15552025-04-30T06:35:32.885ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "187db75f-b410-4884-87ef-68d4037add9f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:32 GMT"} })
15562025-04-30T06:35:32.885ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15572025-04-30T06:35:32.885ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15582025-04-30T06:35:32.885ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15592025-04-30T06:35:32.885ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15602025-04-30T06:35:32.886ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15612025-04-30T06:35:32.889ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15622025-04-30T06:35:32.893ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15632025-04-30T06:35:32.894ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
15642025-04-30T06:35:32.894ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15652025-04-30T06:35:32.894ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15662025-04-30T06:35:32.894ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15672025-04-30T06:35:32.894ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15682025-04-30T06:35:32.903ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15692025-04-30T06:35:32.910ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
15702025-04-30T06:35:32.910ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15712025-04-30T06:35:32.910ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15722025-04-30T06:35:32.910ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15732025-04-30T06:35:32.992ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
15742025-04-30T06:35:32.992ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
15752025-04-30T06:35:32.993ZINFOtest_metrics (InternalApi): Inserted networking records for nexus service
15762025-04-30T06:35:32.993ZINFOtest_metrics (InternalApi): No networking records needed for clickhouse service
15772025-04-30T06:35:32.993ZINFOtest_metrics (InternalApi): No networking records needed for crucible_pantry service
15782025-04-30T06:35:32.993ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:51932 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
15792025-04-30T06:35:32.994ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:51932 req_id = 73264b48-39ea-4eb6-82ee-13706c586fcf sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15802025-04-30T06:35:32.994ZINFOtest_metrics (dropshot): request completed latency_us = 509 local_addr = [::1]:56237 method = GET remote_addr = [::1]:51932 req_id = 73264b48-39ea-4eb6-82ee-13706c586fcf response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
15812025-04-30T06:35:32.995ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "73264b48-39ea-4eb6-82ee-13706c586fcf", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:35:32 GMT"} })
15822025-04-30T06:35:32.997ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15832025-04-30T06:35:32.997ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
15842025-04-30T06:35:32.997ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
15852025-04-30T06:35:32.999ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
15862025-04-30T06:35:32.999ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
15872025-04-30T06:35:32.999ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15882025-04-30T06:35:32.999ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15892025-04-30T06:35:33.002ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15902025-04-30T06:35:33.005ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
15912025-04-30T06:35:33.010ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
15922025-04-30T06:35:33.013ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
15932025-04-30T06:35:33.013ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
15942025-04-30T06:35:33.015ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
15952025-04-30T06:35:33.025ZINFOtest_metrics (InternalApi): Inserted networking records for external_dns service
15962025-04-30T06:35:33.025ZINFOtest_metrics (InternalApi): Inserted service networking records
15972025-04-30T06:35:33.025ZINFOtest_metrics (InternalApi): Inserted physical disks
15982025-04-30T06:35:33.025ZINFOtest_metrics (InternalApi): Inserted zpools
15992025-04-30T06:35:33.025ZINFOtest_metrics (InternalApi): Inserted datasets
16002025-04-30T06:35:33.044ZINFOtest_metrics (InternalApi): Populated DNS tables for internal DNS
16012025-04-30T06:35:33.051ZINFOtest_metrics (InternalApi): Populated DNS tables for external DNS
16022025-04-30T06:35:33.051ZTRACtest_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16032025-04-30T06:35:33.051ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16042025-04-30T06:35:33.051ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16052025-04-30T06:35:33.051ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16062025-04-30T06:35:33.052ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16072025-04-30T06:35:33.056ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16082025-04-30T06:35:33.062ZDEBGtest_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16092025-04-30T06:35:33.063ZTRACtest_metrics (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16102025-04-30T06:35:33.063ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16112025-04-30T06:35:33.063ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16122025-04-30T06:35:33.063ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16132025-04-30T06:35:33.064ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16142025-04-30T06:35:33.080ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16152025-04-30T06:35:33.089ZDEBGtest_metrics (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16162025-04-30T06:35:33.145ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16172025-04-30T06:35:33.145ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16182025-04-30T06:35:33.145ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16192025-04-30T06:35:33.145ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16202025-04-30T06:35:33.146ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16212025-04-30T06:35:33.151ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16222025-04-30T06:35:33.163ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16232025-04-30T06:35:33.163ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16242025-04-30T06:35:33.163ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16252025-04-30T06:35:33.163ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16262025-04-30T06:35:33.163ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16272025-04-30T06:35:33.165ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16282025-04-30T06:35:33.179ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16292025-04-30T06:35:33.192ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16302025-04-30T06:35:33.196ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig
16312025-04-30T06:35:33.196ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16322025-04-30T06:35:33.196ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16332025-04-30T06:35:33.196ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16342025-04-30T06:35:33.197ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16352025-04-30T06:35:33.208ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16362025-04-30T06:35:33.241ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = DnsConfig result = Ok(())
16372025-04-30T06:35:33.257ZINFOtest_metrics (InternalApi): Created recovery silo
16382025-04-30T06:35:33.259ZINFOtest_metrics (InternalApi): Created recovery user
16392025-04-30T06:35:33.264ZINFOtest_metrics (InternalApi): Created recovery user's password
16402025-04-30T06:35:33.264ZTRACtest_metrics (InternalApi): authorize begin action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 76878a9c-e64e-4dd0-9748-c6cec69e7011, lookup_type: ById(76878a9c-e64e-4dd0-9748-c6cec69e7011) }
16412025-04-30T06:35:33.264ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 76878a9c-e64e-4dd0-9748-c6cec69e7011 resource_type = Silo
16422025-04-30T06:35:33.264ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16432025-04-30T06:35:33.264ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16442025-04-30T06:35:33.265ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16452025-04-30T06:35:33.272ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16462025-04-30T06:35:33.272ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16472025-04-30T06:35:33.272ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16482025-04-30T06:35:33.273ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16492025-04-30T06:35:33.286ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16502025-04-30T06:35:33.290ZDEBGtest_metrics (InternalApi): authorize result action = ModifyPolicy actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Silo { parent: Fleet, key: 76878a9c-e64e-4dd0-9748-c6cec69e7011, lookup_type: ById(76878a9c-e64e-4dd0-9748-c6cec69e7011) } result = Ok(())
16512025-04-30T06:35:33.290ZDEBGtest_metrics (InternalApi): Generated role assignment queries
16522025-04-30T06:35:33.297ZINFOtest_metrics (InternalApi): Granted Silo privileges
16532025-04-30T06:35:33.298ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
16542025-04-30T06:35:33.298ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16552025-04-30T06:35:33.298ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16562025-04-30T06:35:33.298ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16572025-04-30T06:35:33.300ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16582025-04-30T06:35:33.303ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16592025-04-30T06:35:33.309ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
16602025-04-30T06:35:33.326ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16612025-04-30T06:35:33.326ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16622025-04-30T06:35:33.327ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16632025-04-30T06:35:33.336ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16642025-04-30T06:35:33.337ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16652025-04-30T06:35:33.337ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16662025-04-30T06:35:33.349ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16672025-04-30T06:35:33.349ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16682025-04-30T06:35:33.350ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16692025-04-30T06:35:33.359ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16702025-04-30T06:35:33.359ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16712025-04-30T06:35:33.359ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16722025-04-30T06:35:33.359ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16732025-04-30T06:35:33.359ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16742025-04-30T06:35:33.371ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16752025-04-30T06:35:33.371ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16762025-04-30T06:35:33.371ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16772025-04-30T06:35:33.372ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16782025-04-30T06:35:33.381ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16792025-04-30T06:35:33.381ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16802025-04-30T06:35:33.382ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16812025-04-30T06:35:33.383ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16822025-04-30T06:35:33.391ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16832025-04-30T06:35:33.409ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
16842025-04-30T06:35:33.410ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
16852025-04-30T06:35:33.410ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
16862025-04-30T06:35:33.410ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
16872025-04-30T06:35:33.410ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16882025-04-30T06:35:33.410ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16892025-04-30T06:35:33.410ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16902025-04-30T06:35:33.422ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
16912025-04-30T06:35:33.423ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16922025-04-30T06:35:33.423ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16932025-04-30T06:35:33.424ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16942025-04-30T06:35:33.427ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
16952025-04-30T06:35:33.428ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
16962025-04-30T06:35:33.428ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
16972025-04-30T06:35:33.429ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
16982025-04-30T06:35:33.435ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
16992025-04-30T06:35:33.445ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17002025-04-30T06:35:33.447ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17012025-04-30T06:35:33.447ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17022025-04-30T06:35:33.448ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17032025-04-30T06:35:33.451ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17042025-04-30T06:35:33.451ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17052025-04-30T06:35:33.451ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17062025-04-30T06:35:33.464ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17072025-04-30T06:35:33.464ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17082025-04-30T06:35:33.465ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17092025-04-30T06:35:33.468ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17102025-04-30T06:35:33.468ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17112025-04-30T06:35:33.469ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17122025-04-30T06:35:33.472ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
17132025-04-30T06:35:33.472ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17142025-04-30T06:35:33.472ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17152025-04-30T06:35:33.472ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17162025-04-30T06:35:33.473ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17172025-04-30T06:35:33.483ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17182025-04-30T06:35:33.483ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17192025-04-30T06:35:33.483ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17202025-04-30T06:35:33.484ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17212025-04-30T06:35:33.486ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17222025-04-30T06:35:33.487ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17232025-04-30T06:35:33.487ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17242025-04-30T06:35:33.487ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17252025-04-30T06:35:33.491ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17262025-04-30T06:35:33.502ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
17272025-04-30T06:35:33.502ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
17282025-04-30T06:35:33.502ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17292025-04-30T06:35:33.502ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17302025-04-30T06:35:33.503ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17312025-04-30T06:35:33.509ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17322025-04-30T06:35:33.509ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17332025-04-30T06:35:33.510ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17342025-04-30T06:35:33.510ZINFOtest_metrics (BackgroundTasks): inserted inventory collection background_task = inventory_collection collection_id = 2faaadaa-5e50-408d-a18e-243c0932f65f
17352025-04-30T06:35:33.510ZDEBGtest_metrics (BackgroundTasks): inventory collection complete background_task = inventory_collection collection_id = 2faaadaa-5e50-408d-a18e-243c0932f65f time_started = 2025-04-30 06:35:32.153283 UTC
17362025-04-30T06:35:33.511ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 4.945149736s iteration = 2 status = {"collection_id":"2faaadaa-5e50-408d-a18e-243c0932f65f","time_done":"2025-04-30 06:35:32.997444 UTC","time_started":"2025-04-30 06:35:32.153283 UTC"}
17372025-04-30T06:35:33.511ZDEBGtest_metrics (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 2 reason = Dependency
17382025-04-30T06:35:33.511ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 43.336\u{b5}s iteration = 2 status = {"error":"task disabled"}
17392025-04-30T06:35:33.511ZDEBGtest_metrics (BackgroundTasks): activating background_task = blueprint_rendezvous iteration = 2 reason = Dependency
17402025-04-30T06:35:33.511ZWARNtest_metrics (BackgroundTasks): Blueprint rendezvous: skipped background_task = blueprint_rendezvous reason = no blueprint
17412025-04-30T06:35:33.511ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = blueprint_rendezvous elapsed = 17.415\u{b5}s iteration = 2 status = {"error":"no blueprint"}
17422025-04-30T06:35:33.515ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17432025-04-30T06:35:33.515ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17442025-04-30T06:35:33.516ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17452025-04-30T06:35:33.522ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17462025-04-30T06:35:33.522ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17472025-04-30T06:35:33.523ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17482025-04-30T06:35:33.540ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17492025-04-30T06:35:33.540ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17502025-04-30T06:35:33.540ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17512025-04-30T06:35:33.540ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17522025-04-30T06:35:33.541ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17532025-04-30T06:35:33.548ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17542025-04-30T06:35:33.548ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17552025-04-30T06:35:33.548ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17562025-04-30T06:35:33.549ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17572025-04-30T06:35:33.552ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17582025-04-30T06:35:33.552ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17592025-04-30T06:35:33.552ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17602025-04-30T06:35:33.553ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17612025-04-30T06:35:33.557ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17622025-04-30T06:35:33.566ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17632025-04-30T06:35:33.566ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
17642025-04-30T06:35:33.566ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
17652025-04-30T06:35:33.566ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17662025-04-30T06:35:33.566ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17672025-04-30T06:35:33.566ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17682025-04-30T06:35:33.567ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17692025-04-30T06:35:33.575ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
17702025-04-30T06:35:33.575ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17712025-04-30T06:35:33.575ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17722025-04-30T06:35:33.578ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17732025-04-30T06:35:33.583ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
17742025-04-30T06:35:33.583ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17752025-04-30T06:35:33.585ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17762025-04-30T06:35:33.587ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17772025-04-30T06:35:33.626ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
17782025-04-30T06:35:33.635ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
17792025-04-30T06:35:33.636ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17802025-04-30T06:35:33.636ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17812025-04-30T06:35:33.636ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17822025-04-30T06:35:33.665ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17832025-04-30T06:35:33.665ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17842025-04-30T06:35:33.666ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17852025-04-30T06:35:33.679ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17862025-04-30T06:35:33.679ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17872025-04-30T06:35:33.681ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17882025-04-30T06:35:33.696ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17892025-04-30T06:35:33.696ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17902025-04-30T06:35:33.697ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17912025-04-30T06:35:33.703ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17922025-04-30T06:35:33.703ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17932025-04-30T06:35:33.704ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17942025-04-30T06:35:33.709ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
17952025-04-30T06:35:33.709ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
17962025-04-30T06:35:33.709ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
17972025-04-30T06:35:33.709ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
17982025-04-30T06:35:33.710ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
17992025-04-30T06:35:33.718ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18002025-04-30T06:35:33.718ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18012025-04-30T06:35:33.718ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18022025-04-30T06:35:33.719ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18032025-04-30T06:35:33.722ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18042025-04-30T06:35:33.723ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18052025-04-30T06:35:33.723ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18062025-04-30T06:35:33.723ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18072025-04-30T06:35:33.739ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18082025-04-30T06:35:33.748ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18092025-04-30T06:35:33.757ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
18102025-04-30T06:35:33.757ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
18112025-04-30T06:35:33.757ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
18122025-04-30T06:35:33.757ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18132025-04-30T06:35:33.758ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18142025-04-30T06:35:33.760ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18152025-04-30T06:35:33.762ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
18162025-04-30T06:35:33.762ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18172025-04-30T06:35:33.762ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18182025-04-30T06:35:33.762ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18192025-04-30T06:35:33.788ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18202025-04-30T06:35:33.788ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18212025-04-30T06:35:33.788ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18222025-04-30T06:35:33.789ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18232025-04-30T06:35:33.799ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18242025-04-30T06:35:33.807ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
18252025-04-30T06:35:33.807ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18262025-04-30T06:35:33.807ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18272025-04-30T06:35:33.808ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18282025-04-30T06:35:33.829ZDEBGtest_metrics (nexus): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("external-dns"): [NetworkInterface { id: 0671dab9-fd2e-4608-bc87-2dab839ac88f, kind: Service { id: d51ab00e-a3ad-4cf7-984a-52d44e99a71f }, name: Name("external-dns-d51ab00e-a3ad-4cf7-984a-52d44e99a71f"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("nexus"): [NetworkInterface { id: b0e1a047-e025-4fdc-a488-6ad3731a6e2b, kind: Service { id: 588db90d-5640-4be3-9102-2ebf0af0c9a3 }, name: Name("nexus-588db90d-5640-4be3-9102-2ebf0af0c9a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
18292025-04-30T06:35:33.829ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
18302025-04-30T06:35:33.829ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18312025-04-30T06:35:33.830ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18322025-04-30T06:35:33.830ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18332025-04-30T06:35:33.830ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18342025-04-30T06:35:33.832ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
18352025-04-30T06:35:33.836ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
18362025-04-30T06:35:33.836ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
18372025-04-30T06:35:33.836ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
18382025-04-30T06:35:33.838ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
18392025-04-30T06:35:33.841ZTRACtest_metrics (nexus): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:35:33.309753Z, allowed_ips: None }
18402025-04-30T06:35:33.841ZDEBGtest_metrics (nexus): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
18412025-04-30T06:35:33.842ZDEBGtest_metrics (nexus): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 0671dab9-fd2e-4608-bc87-2dab839ac88f, kind: Service { id: d51ab00e-a3ad-4cf7-984a-52d44e99a71f }, name: Name("external-dns-d51ab00e-a3ad-4cf7-984a-52d44e99a71f"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: b0e1a047-e025-4fdc-a488-6ad3731a6e2b, kind: Service { id: 588db90d-5640-4be3-9102-2ebf0af0c9a3 }, name: Name("nexus-588db90d-5640-4be3-9102-2ebf0af0c9a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
18422025-04-30T06:35:33.842ZDEBGtest_metrics (nexus): resolved 2 rules for sleds
18432025-04-30T06:35:33.886ZDEBGtest_metrics (nexus): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:35:27.434010Z, time_modified: 2025-04-30T06:35:27.434010Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(56237), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(54162) }]
18442025-04-30T06:35:33.886ZDEBGtest_metrics (nexus): sending firewall rules to sled agents
18452025-04-30T06:35:33.886ZTRACtest_metrics (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18462025-04-30T06:35:33.886ZDEBGtest_metrics (InstanceAllocator): roles roles = RoleSet { roles: {} }
18472025-04-30T06:35:33.887ZDEBGtest_metrics (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18482025-04-30T06:35:33.899ZTRACtest_metrics (InstanceAllocator): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
18492025-04-30T06:35:33.899ZTRACtest_metrics (InstanceAllocator): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18502025-04-30T06:35:33.899ZTRACtest_metrics (InstanceAllocator): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database
18512025-04-30T06:35:33.899ZDEBGtest_metrics (InstanceAllocator): roles roles = RoleSet { roles: {} }
18522025-04-30T06:35:33.899ZDEBGtest_metrics (InstanceAllocator): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Database result = Ok(())
18532025-04-30T06:35:33.908ZDEBGtest_metrics (InstanceAllocator): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "viewer")} }
18542025-04-30T06:35:33.912ZDEBGtest_metrics (InstanceAllocator): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000004ead, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
18552025-04-30T06:35:33.980ZDEBGtest_metrics (nexus): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:56237/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18562025-04-30T06:35:33.981ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:47044 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
18572025-04-30T06:35:33.983ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:47044 req_id = 7eed6375-9b44-4c5e-9d67-1cfc7e2d8834 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18582025-04-30T06:35:33.985ZINFOtest_metrics (dropshot): request completed latency_us = 1961 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:47044 req_id = 7eed6375-9b44-4c5e-9d67-1cfc7e2d8834 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
18592025-04-30T06:35:33.985ZDEBGtest_metrics (nexus): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:56237/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "7eed6375-9b44-4c5e-9d67-1cfc7e2d8834", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
18602025-04-30T06:35:33.986ZINFOtest_metrics (nexus): updated firewall rules on 1 sleds
18612025-04-30T06:35:33.987ZTRACtest_metrics (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
18622025-04-30T06:35:33.987ZDEBGtest_metrics (ServiceBalancer): roles roles = RoleSet { roles: {} }
18632025-04-30T06:35:33.987ZDEBGtest_metrics (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
18642025-04-30T06:35:33.988ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_config_internal dns_group = internal iteration = 2 reason = Signaled
18652025-04-30T06:35:33.988ZDEBGtest_metrics (BackgroundTasks): reading DNS version background_task = dns_config_internal dns_group = internal
18662025-04-30T06:35:33.988ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
18672025-04-30T06:35:33.988ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
18682025-04-30T06:35:33.988ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
18692025-04-30T06:35:33.988ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_servers_internal dns_group = internal iteration = 2 reason = Signaled
18702025-04-30T06:35:33.988ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal
18712025-04-30T06:35:33.988ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_config_external dns_group = external iteration = 2 reason = Signaled
18722025-04-30T06:35:33.989ZDEBGtest_metrics (BackgroundTasks): reading DNS version background_task = dns_config_external dns_group = external
18732025-04-30T06:35:33.989ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
18742025-04-30T06:35:33.989ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
18752025-04-30T06:35:33.989ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
18762025-04-30T06:35:33.989ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_servers_external dns_group = external iteration = 2 reason = Signaled
18772025-04-30T06:35:33.989ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal
18782025-04-30T06:35:33.989ZDEBGtest_metrics (BackgroundTasks): activating background_task = external_endpoints iteration = 2 reason = Signaled
18792025-04-30T06:35:33.989ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
18802025-04-30T06:35:33.989ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18812025-04-30T06:35:33.989ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
18822025-04-30T06:35:33.989ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
18832025-04-30T06:35:33.990ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
18842025-04-30T06:35:33.990ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 3 reason = Signaled
18852025-04-30T06:35:33.990ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
18862025-04-30T06:35:33.990ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
18872025-04-30T06:35:33.990ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
18882025-04-30T06:35:33.990ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
18892025-04-30T06:35:33.990ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
18902025-04-30T06:35:33.991ZTRACtest_metrics (dns): buffer buffer = Length: 75 (0x4b) bytes\n0000: c4 82 01 00 00 01 00 00 00 00 00 01 0c 5f 6e 61 ............._na\n0010: 6d 65 73 65 72 76 69 63 65 04 5f 74 63 70 0d 63 meservice._tcp.c\n0020: 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 ontrol-plane.oxi\n0030: 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 de.internal..!..\n0040: 00 00 29 04 d0 00 00 00 00 00 00 ..)........ kind = dns peer_addr = [::1]:61000 req_id = fd4d0cdc-2277-4648-82af-f1fd50f18590
18912025-04-30T06:35:33.991ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50306,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_nameservice._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_nameservice._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 12,\n 95,\n 110,\n 97,\n 109,\n 101,\n 115,\n 101,\n 114,\n 118,\n 105,\n 99,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61000 req_id = fd4d0cdc-2277-4648-82af-f1fd50f18590
18922025-04-30T06:35:33.992ZDEBGtest_metrics (store): query key key = _nameservice._tcp
18932025-04-30T06:35:33.992ZDEBGtest_metrics (store): query key key = 441da737-6ecf-40f8-95fb-d6fbdea9e50e.host
18942025-04-30T06:35:33.992ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61000 query = LowerQuery { name: LowerName(Name("_nameservice._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36868, target: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal") })) }] req_id = fd4d0cdc-2277-4648-82af-f1fd50f18590
18952025-04-30T06:35:33.992ZTRACtest_metrics (dns): buffer buffer = Length: 76 (0x4c) bytes\n0000: 28 86 01 00 00 01 00 00 00 00 00 01 0d 5f 65 78 (............_ex\n0010: 74 65 72 6e 61 6c 2d 64 6e 73 04 5f 74 63 70 0d ternal-dns._tcp.\n0020: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0030: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 21 00 ide.internal..!.\n0040: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57308 req_id = d5f9931d-6ef7-4da1-bd01-568926db1c75
18962025-04-30T06:35:33.992ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10374,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_external-dns._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_external-dns._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 13,\n 95,\n 101,\n 120,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 45,\n 100,\n 110,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57308 req_id = d5f9931d-6ef7-4da1-bd01-568926db1c75
18972025-04-30T06:35:33.992ZDEBGtest_metrics (store): query key key = _external-dns._tcp
18982025-04-30T06:35:33.992ZDEBGtest_metrics (store): query key key = d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host
18992025-04-30T06:35:33.993ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57308 query = LowerQuery { name: LowerName(Name("_external-dns._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 43730, target: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal") })) }] req_id = d5f9931d-6ef7-4da1-bd01-568926db1c75
19002025-04-30T06:35:33.994ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _nameservice._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_nameservice._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_nameservice._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36868, target: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3798, tv_nsec: 793170752 } })
19012025-04-30T06:35:33.994ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
19022025-04-30T06:35:33.994ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _external-dns._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_external-dns._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_external-dns._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 43730, target: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3798, tv_nsec: 793695498 } })
19032025-04-30T06:35:33.994ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
19042025-04-30T06:35:33.995ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19052025-04-30T06:35:33.995ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19062025-04-30T06:35:33.995ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19072025-04-30T06:35:33.995ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19082025-04-30T06:35:33.995ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19092025-04-30T06:35:33.996ZTRACtest_metrics (ServiceBalancer): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) }
19102025-04-30T06:35:33.996ZTRACtest_metrics (ServiceBalancer): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19112025-04-30T06:35:33.996ZTRACtest_metrics (ServiceBalancer): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database
19122025-04-30T06:35:33.996ZDEBGtest_metrics (ServiceBalancer): roles roles = RoleSet { roles: {} }
19132025-04-30T06:35:33.997ZDEBGtest_metrics (ServiceBalancer): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Database result = Ok(())
19142025-04-30T06:35:33.997ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19152025-04-30T06:35:33.997ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19162025-04-30T06:35:33.997ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19172025-04-30T06:35:33.997ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19182025-04-30T06:35:33.997ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19192025-04-30T06:35:33.998ZTRACtest_metrics (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 65 1d 01 00 00 01 00 00 00 00 00 01 24 34 34 31 e...........$441\n0010: 64 61 37 33 37 2d 36 65 63 66 2d 34 30 66 38 2d da737-6ecf-40f8-\n0020: 39 35 66 62 2d 64 36 66 62 64 65 61 39 65 35 30 95fb-d6fbdea9e50\n0030: 65 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 e.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:62627 req_id = 09c39f33-dbfd-4eac-a240-e014074d6b56
19202025-04-30T06:35:33.998ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25885,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 52,\n 52,\n 49,\n 100,\n 97,\n 55,\n 51,\n 55,\n 45,\n 54,\n 101,\n 99,\n 102,\n 45,\n 52,\n 48,\n 102,\n 56,\n 45,\n 57,\n 53,\n 102,\n 98,\n 45,\n 100,\n 54,\n 102,\n 98,\n 100,\n 101,\n 97,\n 57,\n 101,\n 53,\n 48,\n 101,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62627 req_id = 09c39f33-dbfd-4eac-a240-e014074d6b56
19212025-04-30T06:35:33.998ZDEBGtest_metrics (store): query key key = 441da737-6ecf-40f8-95fb-d6fbdea9e50e.host
19222025-04-30T06:35:33.998ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62627 query = LowerQuery { name: LowerName(Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal.")), original: Query { name: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("441da737-6ecf-40f8-95fb-d6fbdea9e50e.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 09c39f33-dbfd-4eac-a240-e014074d6b56
19232025-04-30T06:35:33.998ZTRACtest_metrics (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: 64 0b 01 00 00 01 00 00 00 00 00 01 24 64 35 31 d...........$d51\n0010: 61 62 30 30 65 2d 61 33 61 64 2d 34 63 66 37 2d ab00e-a3ad-4cf7-\n0020: 39 38 34 61 2d 35 32 64 34 34 65 39 39 61 37 31 984a-52d44e99a71\n0030: 66 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 f.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:61371 req_id = 2fcd19de-ae10-415b-901e-e362cec230f0
19242025-04-30T06:35:33.998ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25611,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 100,\n 53,\n 49,\n 97,\n 98,\n 48,\n 48,\n 101,\n 45,\n 97,\n 51,\n 97,\n 100,\n 45,\n 52,\n 99,\n 102,\n 55,\n 45,\n 57,\n 56,\n 52,\n 97,\n 45,\n 53,\n 50,\n 100,\n 52,\n 52,\n 101,\n 57,\n 57,\n 97,\n 55,\n 49,\n 102,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61371 req_id = 2fcd19de-ae10-415b-901e-e362cec230f0
19252025-04-30T06:35:33.998ZDEBGtest_metrics (store): query key key = d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host
19262025-04-30T06:35:33.998ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:61371 query = LowerQuery { name: LowerName(Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal.")), original: Query { name: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("d51ab00e-a3ad-4cf7-984a-52d44e99a71f.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2fcd19de-ae10-415b-901e-e362cec230f0
19272025-04-30T06:35:34.001ZINFOtest_metrics (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:36868] } background_task = dns_servers_internal dns_group = internal
19282025-04-30T06:35:34.001ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_servers_internal dns_group = internal elapsed = 12.725926ms iteration = 2 status = {"addresses":["[::1]:36868"]}
19292025-04-30T06:35:34.001ZINFOtest_metrics (BackgroundTasks): found DNS servers (initial) addresses = DnsServersList { addresses: [[::1]:43730] } background_task = dns_servers_external dns_group = external
19302025-04-30T06:35:34.001ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_servers_external dns_group = external elapsed = 12.182503ms iteration = 2 status = {"addresses":["[::1]:43730"]}
19312025-04-30T06:35:34.001ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 2 reason = Dependency
19322025-04-30T06:35:34.001ZWARNtest_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_internal dns_group = internal reason = no config
19332025-04-30T06:35:34.001ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 20.521\u{b5}s iteration = 2 status = {"error":"no config"}
19342025-04-30T06:35:34.001ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 2 reason = Dependency
19352025-04-30T06:35:34.001ZWARNtest_metrics (BackgroundTasks): DNS propagation: skipped background_task = dns_propagation_external dns_group = external reason = no config
19362025-04-30T06:35:34.001ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 9.729\u{b5}s iteration = 2 status = {"error":"no config"}
19372025-04-30T06:35:34.002ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19382025-04-30T06:35:34.006ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
19392025-04-30T06:35:34.006ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19402025-04-30T06:35:34.006ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19412025-04-30T06:35:34.007ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19422025-04-30T06:35:34.007ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19432025-04-30T06:35:34.025ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
19442025-04-30T06:35:34.025ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
19452025-04-30T06:35:34.025ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
19462025-04-30T06:35:34.026ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
19472025-04-30T06:35:34.028ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19482025-04-30T06:35:34.041ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19492025-04-30T06:35:34.043ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19502025-04-30T06:35:34.057ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
19512025-04-30T06:35:34.123ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig
19522025-04-30T06:35:34.123ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19532025-04-30T06:35:34.123ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19542025-04-30T06:35:34.123ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19552025-04-30T06:35:34.124ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19562025-04-30T06:35:34.125ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
19572025-04-30T06:35:34.125ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
19582025-04-30T06:35:34.126ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 20 dc 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58029 req_id = 817a8106-9f31-431d-a64d-7da60928d926
19592025-04-30T06:35:34.126ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8412,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58029 req_id = 817a8106-9f31-431d-a64d-7da60928d926
19602025-04-30T06:35:34.126ZDEBGtest_metrics (store): query key key = _mgs._tcp
19612025-04-30T06:35:34.127ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19622025-04-30T06:35:34.127ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58029 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 817a8106-9f31-431d-a64d-7da60928d926
19632025-04-30T06:35:34.128ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3798, tv_nsec: 926999858 } })
19642025-04-30T06:35:34.128ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
19652025-04-30T06:35:34.129ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: e3 5d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .]..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60020 req_id = 6ccad8b4-5538-4055-b6b6-0bc76c93d43c
19662025-04-30T06:35:34.129ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58205,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60020 req_id = 6ccad8b4-5538-4055-b6b6-0bc76c93d43c
19672025-04-30T06:35:34.129ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
19682025-04-30T06:35:34.130ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60020 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6ccad8b4-5538-4055-b6b6-0bc76c93d43c
19692025-04-30T06:35:34.249ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
19702025-04-30T06:35:34.249ZDEBGtest_metrics (BackgroundTasks): found DNS version background_task = dns_config_internal dns_group = internal version = 1
19712025-04-30T06:35:34.250ZDEBGtest_metrics (BackgroundTasks): reading DNS config background_task = dns_config_internal dns_group = internal
19722025-04-30T06:35:34.250ZDEBGtest_metrics (BackgroundTasks): reading DNS zones background_task = dns_config_internal dns_group = internal
19732025-04-30T06:35:34.250ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
19742025-04-30T06:35:34.251ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19752025-04-30T06:35:34.251ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
19762025-04-30T06:35:34.251ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
19772025-04-30T06:35:34.253ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
19782025-04-30T06:35:34.253ZDEBGtest_metrics (BackgroundTasks): found DNS version background_task = dns_config_external dns_group = external version = 2
19792025-04-30T06:35:34.253ZDEBGtest_metrics (BackgroundTasks): reading DNS config background_task = dns_config_external dns_group = external
19802025-04-30T06:35:34.253ZDEBGtest_metrics (BackgroundTasks): reading DNS zones background_task = dns_config_external dns_group = external
19812025-04-30T06:35:34.253ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
19822025-04-30T06:35:34.253ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
19832025-04-30T06:35:34.253ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
19842025-04-30T06:35:34.253ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
19852025-04-30T06:35:34.254ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
19862025-04-30T06:35:34.254ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19872025-04-30T06:35:34.260ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = DnsConfig result = Ok(())
19882025-04-30T06:35:34.260ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
19892025-04-30T06:35:34.260ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
19902025-04-30T06:35:34.261ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
19912025-04-30T06:35:34.262ZDEBGtest_metrics (ServiceBalancer): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19922025-04-30T06:35:34.272ZDEBGtest_metrics (ServiceBalancer): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-00000000bac3, .. }) resource = Rack { parent: Fleet, key: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, lookup_type: ById(c19a698f-c6f9-4a17-ae30-20d711b8f7dc) } result = Ok(())
19932025-04-30T06:35:34.272ZINFOtest_metrics (nexus): Rack initialized
19942025-04-30T06:35:34.272ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
19952025-04-30T06:35:34.272ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
19962025-04-30T06:35:34.273ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
19972025-04-30T06:35:34.275ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
19982025-04-30T06:35:34.294ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
19992025-04-30T06:35:34.294ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20002025-04-30T06:35:34.294ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20012025-04-30T06:35:34.295ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20022025-04-30T06:35:34.295ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20032025-04-30T06:35:34.301ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20042025-04-30T06:35:34.301ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20052025-04-30T06:35:34.301ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20062025-04-30T06:35:34.308ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20072025-04-30T06:35:34.308ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet
20082025-04-30T06:35:34.308ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = external_endpoints resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20092025-04-30T06:35:34.308ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20102025-04-30T06:35:34.308ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20112025-04-30T06:35:34.309ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20122025-04-30T06:35:34.310ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 13 a8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60363 req_id = 519ff2f3-d1c0-4dec-9a9c-e331c7497108
20132025-04-30T06:35:34.310ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5032,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60363 req_id = 519ff2f3-d1c0-4dec-9a9c-e331c7497108
20142025-04-30T06:35:34.310ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
20152025-04-30T06:35:34.310ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5032, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60363 req_id = 519ff2f3-d1c0-4dec-9a9c-e331c7497108
20162025-04-30T06:35:34.311ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
20172025-04-30T06:35:34.311ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
20182025-04-30T06:35:34.311ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
20192025-04-30T06:35:34.313ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20202025-04-30T06:35:34.313ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20212025-04-30T06:35:34.315ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20222025-04-30T06:35:34.315ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad3c5d8b-2717-4776-8910-a1351f8591f6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
20232025-04-30T06:35:34.316ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
20242025-04-30T06:35:34.317ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20252025-04-30T06:35:34.324ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Fleet result = Ok(())
20262025-04-30T06:35:34.324ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database
20272025-04-30T06:35:34.324ZDEBGtest_metrics (BackgroundTasks): roles background_task = external_endpoints roles = RoleSet { roles: {} }
20282025-04-30T06:35:34.325ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = external_endpoints resource = Database result = Ok(())
20292025-04-30T06:35:34.325ZDEBGtest_metrics (BackgroundTasks): found zones background_task = dns_config_external count = 1 dns_group = external
20302025-04-30T06:35:34.325ZDEBGtest_metrics (BackgroundTasks): listing DNS names for zone background_task = dns_config_external batch_size = 100 dns_group = external dns_zone_id = 716b4d1c-6744-4256-be7d-20432fe47fb1 dns_zone_name = oxide-dev.test found_so_far = 0 version = 2
20312025-04-30T06:35:34.325ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig
20322025-04-30T06:35:34.325ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_external dns_group = external resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20332025-04-30T06:35:34.325ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20342025-04-30T06:35:34.326ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20352025-04-30T06:35:34.326ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20362025-04-30T06:35:34.345ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20372025-04-30T06:35:34.366ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = DnsConfig result = Ok(())
20382025-04-30T06:35:34.366ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database
20392025-04-30T06:35:34.366ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_external dns_group = external roles = RoleSet { roles: {} }
20402025-04-30T06:35:34.367ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_external dns_group = external resource = Database result = Ok(())
20412025-04-30T06:35:34.367ZDEBGtest_metrics (BackgroundTasks): found zones background_task = dns_config_internal count = 1 dns_group = internal
20422025-04-30T06:35:34.367ZDEBGtest_metrics (BackgroundTasks): listing DNS names for zone background_task = dns_config_internal batch_size = 100 dns_group = internal dns_zone_id = eec12680-aa0d-4433-992b-efd6c7c68f17 dns_zone_name = control-plane.oxide.internal found_so_far = 0 version = 1
20432025-04-30T06:35:34.367ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig
20442025-04-30T06:35:34.368ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = dns_config_internal dns_group = internal resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
20452025-04-30T06:35:34.368ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20462025-04-30T06:35:34.368ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20472025-04-30T06:35:34.368ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20482025-04-30T06:35:34.369ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20492025-04-30T06:35:34.369ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20502025-04-30T06:35:34.370ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20512025-04-30T06:35:34.390ZINFOtest_metrics (BackgroundTasks): found Silo/DNS/TLS config (initial) background_task = external_endpoints config = ExternalEndpoints { by_dns_name: {"default-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 001de000-5110-4000-8000-000000000000, db_silo: Silo { identity: SiloIdentity { id: 001de000-5110-4000-8000-000000000000, name: Name(Name("default-silo")), description: "default silo", time_created: 2025-04-30T06:35:25.687092Z, time_modified: 2025-04-30T06:35:25.687092Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {}, rcgen: Generation(Generation(1)) }, tls_certs: [] }, "test-suite-silo.sys.oxide-dev.test": ExternalEndpoint { silo_id: 76878a9c-e64e-4dd0-9748-c6cec69e7011, db_silo: Silo { identity: SiloIdentity { id: 76878a9c-e64e-4dd0-9748-c6cec69e7011, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:35:33.051663Z, time_modified: 2025-04-30T06:35:33.051663Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }}, warnings: [NoSiloCerts { silo_id: 001de000-5110-4000-8000-000000000000, dns_name: "default-silo.sys.oxide-dev.test" }, NoSiloCerts { silo_id: 76878a9c-e64e-4dd0-9748-c6cec69e7011, dns_name: "test-suite-silo.sys.oxide-dev.test" }], default_endpoint: Some(ExternalEndpoint { silo_id: 76878a9c-e64e-4dd0-9748-c6cec69e7011, db_silo: Silo { identity: SiloIdentity { id: 76878a9c-e64e-4dd0-9748-c6cec69e7011, name: Name(Name("test-suite-silo")), description: "built-in recovery Silo", time_created: 2025-04-30T06:35:33.051663Z, time_modified: 2025-04-30T06:35:33.051663Z, time_deleted: None }, discoverable: false, authentication_mode: Local, user_provision_type: ApiOnly, mapped_fleet_roles: Object {"admin": Array [String("admin")]}, rcgen: Generation(Generation(1)) }, tls_certs: [] }) }
20522025-04-30T06:35:34.390ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = external_endpoints elapsed = 400.879807ms iteration = 2 status = {"by_dns_name":{"default-silo.sys.oxide-dev.test":{"silo_id":"001de000-5110-4000-8000-000000000000","tls_certs":[]},"test-suite-silo.sys.oxide-dev.test":{"silo_id":"76878a9c-e64e-4dd0-9748-c6cec69e7011","tls_certs":[]}},"default_endpoint":{"silo_id":"76878a9c-e64e-4dd0-9748-c6cec69e7011","tls_certs":[]},"warnings":["silo 001de000-5110-4000-8000-000000000000 with DNS name \\"default-silo.sys.oxide-dev.test\\" has no usable certificates","silo 76878a9c-e64e-4dd0-9748-c6cec69e7011 with DNS name \\"test-suite-silo.sys.oxide-dev.test\\" has no usable certificates"]}
20532025-04-30T06:35:34.390ZDEBGtest_metrics (BackgroundTasks): found all DNS names for zone background_task = dns_config_external dns_group = external dns_zone_id = 716b4d1c-6744-4256-be7d-20432fe47fb1 dns_zone_name = oxide-dev.test found_so_far = 1 version = 2
20542025-04-30T06:35:34.390ZDEBGtest_metrics (BackgroundTasks): read DNS config background_task = dns_config_external dns_group = external nzones = 1 version = 2
20552025-04-30T06:35:34.390ZINFOtest_metrics (BackgroundTasks): found latest generation (first find) background_task = dns_config_external dns_group = external generation = 2
20562025-04-30T06:35:34.390ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_config_external dns_group = external elapsed = 401.836613ms iteration = 2 status = {"generation":2}
20572025-04-30T06:35:34.390ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_propagation_external dns_group = external iteration = 3 reason = Dependency
20582025-04-30T06:35:34.496ZDEBGtest_metrics (BackgroundTasks): client request background_task = dns_propagation_external body = Some(Body) dns_group = external dns_server_url = http://[::1]:43730 generation = 2 method = PUT servers = DnsServersList { addresses: [[::1]:43730] } uri = http://[::1]:43730/config
20592025-04-30T06:35:34.497ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
20602025-04-30T06:35:34.502ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = DnsConfig result = Ok(())
20612025-04-30T06:35:34.502ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database
20622025-04-30T06:35:34.502ZDEBGtest_metrics (BackgroundTasks): roles background_task = dns_config_internal dns_group = internal roles = RoleSet { roles: {} }
20632025-04-30T06:35:34.502ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = dns_config_internal dns_group = internal resource = Database result = Ok(())
20642025-04-30T06:35:34.503ZINFOtest_metrics (http): accepted connection kind = dns local_addr = [::1]:43730 remote_addr = [::1]:61302
20652025-04-30T06:35:34.507ZTRACtest_metrics (http): incoming request kind = dns local_addr = [::1]:43730 method = PUT remote_addr = [::1]:61302 req_id = d98d61db-d46a-407a-8e03-d63313ab28af uri = /config
20662025-04-30T06:35:34.507ZINFOtest_metrics (store): attempting generation update new_generation = 2 req_id = d98d61db-d46a-407a-8e03-d63313ab28af
20672025-04-30T06:35:34.507ZINFOtest_metrics (store): pruning trees for generations newer than 0
20682025-04-30T06:35:34.507ZDEBGtest_metrics (store): creating tree tree_name = generation_2_zone_oxide-dev.test
20692025-04-30T06:35:34.507ZDEBGtest_metrics (store): updating current config
20702025-04-30T06:35:34.507ZDEBGtest_metrics (store): flushing default tree
20712025-04-30T06:35:34.507ZINFOtest_metrics (store): pruning trees for generations older than 2 keep = 3
20722025-04-30T06:35:34.507ZINFOtest_metrics (store): updated generation new_generation = 2 req_id = d98d61db-d46a-407a-8e03-d63313ab28af
20732025-04-30T06:35:34.508ZINFOtest_metrics (http): request completed kind = dns latency_us = 1029 local_addr = [::1]:43730 method = PUT remote_addr = [::1]:61302 req_id = d98d61db-d46a-407a-8e03-d63313ab28af response_code = 204 uri = /config
20742025-04-30T06:35:34.508ZDEBGtest_metrics (BackgroundTasks): found all DNS names for zone background_task = dns_config_internal dns_group = internal dns_zone_id = eec12680-aa0d-4433-992b-efd6c7c68f17 dns_zone_name = control-plane.oxide.internal found_so_far = 21 version = 1
20752025-04-30T06:35:34.508ZDEBGtest_metrics (BackgroundTasks): read DNS config background_task = dns_config_internal dns_group = internal nzones = 1 version = 1
20762025-04-30T06:35:34.508ZINFOtest_metrics (BackgroundTasks): found latest generation (first find) background_task = dns_config_internal dns_group = internal generation = 1
20772025-04-30T06:35:34.508ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_config_internal dns_group = internal elapsed = 520.358019ms iteration = 2 status = {"generation":1}
20782025-04-30T06:35:34.508ZDEBGtest_metrics (BackgroundTasks): activating background_task = dns_propagation_internal dns_group = internal iteration = 3 reason = Dependency
20792025-04-30T06:35:34.553ZDEBGtest_metrics (BackgroundTasks): client request background_task = dns_propagation_internal body = Some(Body) dns_group = internal dns_server_url = http://[::1]:36868 generation = 1 method = PUT servers = DnsServersList { addresses: [[::1]:36868] } uri = http://[::1]:36868/config
20802025-04-30T06:35:34.554ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
20812025-04-30T06:35:34.554ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
20822025-04-30T06:35:34.554ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
20832025-04-30T06:35:34.554ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
20842025-04-30T06:35:34.554ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
20852025-04-30T06:35:34.555ZINFOtest_metrics (http): accepted connection kind = dns local_addr = [::1]:36868 remote_addr = [::1]:35405
20862025-04-30T06:35:34.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:43730 generation = 2 result = Ok(Response { url: "http://[::1]:43730/config", status: 204, headers: {"x-request-id": "d98d61db-d46a-407a-8e03-d63313ab28af", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} }) servers = DnsServersList { addresses: [[::1]:43730] }
20872025-04-30T06:35:34.557ZINFOtest_metrics (BackgroundTasks): DNS server now at generation 2 background_task = dns_propagation_external dns_group = external dns_server_url = http://[::1]:43730 generation = 2 servers = DnsServersList { addresses: [[::1]:43730] }
20882025-04-30T06:35:34.558ZINFOtest_metrics (BackgroundTasks): DNS propagation: done background_task = dns_propagation_external dns_group = external generation = 2 servers = DnsServersList { addresses: [[::1]:43730] }
20892025-04-30T06:35:34.558ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_propagation_external dns_group = external elapsed = 167.315997ms iteration = 3 status = {"generation":2,"server_results":{"[::1]:43730":{"Ok":null}}}
20902025-04-30T06:35:34.559ZTRACtest_metrics (http): incoming request kind = dns local_addr = [::1]:36868 method = PUT remote_addr = [::1]:35405 req_id = 6b2f8b00-4dc4-4715-97a1-bbd0c2305fc2 uri = /config
20912025-04-30T06:35:34.560ZINFOtest_metrics (store): attempting generation update new_generation = 1 req_id = 6b2f8b00-4dc4-4715-97a1-bbd0c2305fc2
20922025-04-30T06:35:34.560ZINFOtest_metrics (store): updated generation new_generation = 1 req_id = 6b2f8b00-4dc4-4715-97a1-bbd0c2305fc2
20932025-04-30T06:35:34.560ZINFOtest_metrics (http): request completed kind = dns latency_us = 475 local_addr = [::1]:36868 method = PUT remote_addr = [::1]:35405 req_id = 6b2f8b00-4dc4-4715-97a1-bbd0c2305fc2 response_code = 204 uri = /config
20942025-04-30T06:35:34.562ZDEBGtest_metrics (BackgroundTasks): client response background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:36868 generation = 1 result = Ok(Response { url: "http://[::1]:36868/config", status: 204, headers: {"x-request-id": "6b2f8b00-4dc4-4715-97a1-bbd0c2305fc2", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} }) servers = DnsServersList { addresses: [[::1]:36868] }
20952025-04-30T06:35:34.562ZINFOtest_metrics (BackgroundTasks): DNS server now at generation 1 background_task = dns_propagation_internal dns_group = internal dns_server_url = http://[::1]:36868 generation = 1 servers = DnsServersList { addresses: [[::1]:36868] }
20962025-04-30T06:35:34.563ZINFOtest_metrics (BackgroundTasks): DNS propagation: done background_task = dns_propagation_internal dns_group = internal generation = 1 servers = DnsServersList { addresses: [[::1]:36868] }
20972025-04-30T06:35:34.563ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = dns_propagation_internal dns_group = internal elapsed = 55.247276ms iteration = 3 status = {"generation":1,"server_results":{"[::1]:36868":{"Ok":null}}}
20982025-04-30T06:35:34.564ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
20992025-04-30T06:35:34.564ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21002025-04-30T06:35:34.564ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21012025-04-30T06:35:34.564ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21022025-04-30T06:35:34.570ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20675976-7015-47da-b405-db7e7b0d03ae", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21032025-04-30T06:35:34.570ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
21042025-04-30T06:35:34.577ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21052025-04-30T06:35:34.577ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21062025-04-30T06:35:34.577ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21072025-04-30T06:35:34.578ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21082025-04-30T06:35:34.584ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21092025-04-30T06:35:34.597ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21102025-04-30T06:35:34.598ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21112025-04-30T06:35:34.598ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21122025-04-30T06:35:34.598ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21132025-04-30T06:35:34.598ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21142025-04-30T06:35:34.598ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21152025-04-30T06:35:34.599ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21162025-04-30T06:35:34.607ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21172025-04-30T06:35:34.608ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21182025-04-30T06:35:34.608ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21192025-04-30T06:35:34.611ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21202025-04-30T06:35:34.623ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21212025-04-30T06:35:34.623ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21222025-04-30T06:35:34.623ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21232025-04-30T06:35:34.624ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21242025-04-30T06:35:34.630ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b345556-5280-4e7e-8366-762a7af24ece", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21252025-04-30T06:35:34.630ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
21262025-04-30T06:35:34.631ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21272025-04-30T06:35:34.637ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21282025-04-30T06:35:34.637ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21292025-04-30T06:35:34.637ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21302025-04-30T06:35:34.637ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21312025-04-30T06:35:34.641ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2341d8f1-fb23-46fd-943f-488da0ef5b54", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21322025-04-30T06:35:34.642ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
21332025-04-30T06:35:34.642ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21342025-04-30T06:35:34.642ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21352025-04-30T06:35:34.643ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21362025-04-30T06:35:34.645ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21372025-04-30T06:35:34.645ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21382025-04-30T06:35:34.646ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21392025-04-30T06:35:34.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f9a27c0-42d0-4a6d-9a2b-910c40aea083", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21402025-04-30T06:35:34.647ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
21412025-04-30T06:35:34.650ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5dec2d-b648-457a-a8df-c258ae040335", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21422025-04-30T06:35:34.650ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
21432025-04-30T06:35:34.652ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21442025-04-30T06:35:34.652ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21452025-04-30T06:35:34.652ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21462025-04-30T06:35:34.654ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80fc959e-fb8d-40af-9501-c74980e4852f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21472025-04-30T06:35:34.654ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
21482025-04-30T06:35:34.655ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) }
21492025-04-30T06:35:34.655ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21502025-04-30T06:35:34.655ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21512025-04-30T06:35:34.655ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21522025-04-30T06:35:34.656ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21532025-04-30T06:35:34.665ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21542025-04-30T06:35:34.665ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21552025-04-30T06:35:34.665ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21562025-04-30T06:35:34.666ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21572025-04-30T06:35:34.686ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
21582025-04-30T06:35:34.686ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21592025-04-30T06:35:34.686ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21602025-04-30T06:35:34.687ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21612025-04-30T06:35:34.687ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3930814d-18cd-40ad-8dee-d9c97cea00b8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21622025-04-30T06:35:34.688ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
21632025-04-30T06:35:34.690ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "392994ec-ece2-437d-971b-bd0b9f2bfc52", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21642025-04-30T06:35:34.690ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
21652025-04-30T06:35:34.691ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
21662025-04-30T06:35:34.702ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ById(001de000-074c-4000-8000-000000000000) } result = Ok(())
21672025-04-30T06:35:34.702ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "Vpc" using built-in user
21682025-04-30T06:35:34.702ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21692025-04-30T06:35:34.703ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21702025-04-30T06:35:34.703ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21712025-04-30T06:35:34.704ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ad7c0b1-2ddb-43ec-8274-4ddf5852611e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21722025-04-30T06:35:34.704ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
21732025-04-30T06:35:34.705ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21742025-04-30T06:35:34.705ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21752025-04-30T06:35:34.706ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21762025-04-30T06:35:34.707ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e0c28a8-4734-4806-9c84-28bbfeeaaec0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21772025-04-30T06:35:34.707ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
21782025-04-30T06:35:34.709ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21792025-04-30T06:35:34.709ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21802025-04-30T06:35:34.709ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21812025-04-30T06:35:34.709ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4196c025-36c3-492e-8908-174114919589", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21822025-04-30T06:35:34.710ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
21832025-04-30T06:35:34.714ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21842025-04-30T06:35:34.714ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21852025-04-30T06:35:34.715ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21862025-04-30T06:35:34.716ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "30a16480-c2f4-4d27-b5a4-e05423256ed5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21872025-04-30T06:35:34.717ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
21882025-04-30T06:35:34.721ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
21892025-04-30T06:35:34.721ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
21902025-04-30T06:35:34.721ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21912025-04-30T06:35:34.721ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21922025-04-30T06:35:34.723ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21932025-04-30T06:35:34.724ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77b4a35c-30d0-4d25-be50-426f63e26f91", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
21942025-04-30T06:35:34.724ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
21952025-04-30T06:35:34.725ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
21962025-04-30T06:35:34.725ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
21972025-04-30T06:35:34.725ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
21982025-04-30T06:35:34.725ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
21992025-04-30T06:35:34.728ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22002025-04-30T06:35:34.728ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22012025-04-30T06:35:34.728ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22022025-04-30T06:35:34.728ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22032025-04-30T06:35:34.732ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01f3c1cb-38b3-4c9a-b3b7-f17577574d72", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22042025-04-30T06:35:34.733ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
22052025-04-30T06:35:34.733ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22062025-04-30T06:35:34.739ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22072025-04-30T06:35:34.740ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22082025-04-30T06:35:34.740ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") }
22092025-04-30T06:35:34.740ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22102025-04-30T06:35:34.740ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22112025-04-30T06:35:34.740ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22122025-04-30T06:35:34.740ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22132025-04-30T06:35:34.742ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22142025-04-30T06:35:34.742ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22152025-04-30T06:35:34.742ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22162025-04-30T06:35:34.743ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22172025-04-30T06:35:34.743ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a317b5e-1df7-4ee1-94e5-306048233659", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22182025-04-30T06:35:34.743ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
22192025-04-30T06:35:34.745ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22202025-04-30T06:35:34.745ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22212025-04-30T06:35:34.745ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22222025-04-30T06:35:34.746ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22232025-04-30T06:35:34.748ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d652195-67df-4f84-a743-77c08db00e11", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22242025-04-30T06:35:34.748ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
22252025-04-30T06:35:34.748ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22262025-04-30T06:35:34.754ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000001, lookup_type: ByName("external-dns") } result = Ok(())
22272025-04-30T06:35:34.754ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22282025-04-30T06:35:34.754ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22292025-04-30T06:35:34.754ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22302025-04-30T06:35:34.756ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "144ee472-748d-4ae3-862b-26c5da5c803e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22312025-04-30T06:35:34.756ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
22322025-04-30T06:35:34.758ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a9a4ed-fa01-461f-80fe-e075693ee07b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22332025-04-30T06:35:34.758ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
22342025-04-30T06:35:34.759ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e92557d-54f0-4445-a38e-37f141700b27", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22352025-04-30T06:35:34.759ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
22362025-04-30T06:35:34.759ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22372025-04-30T06:35:34.759ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22382025-04-30T06:35:34.760ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22392025-04-30T06:35:34.763ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22402025-04-30T06:35:34.763ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22412025-04-30T06:35:34.764ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22422025-04-30T06:35:34.764ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90425938-6db4-4d5f-a21a-952ab7317c58", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22432025-04-30T06:35:34.766ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
22442025-04-30T06:35:34.768ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66f0a314-1cf0-4647-8f10-d1722f6ce976", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22452025-04-30T06:35:34.768ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
22462025-04-30T06:35:34.768ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22472025-04-30T06:35:34.768ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22482025-04-30T06:35:34.769ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22492025-04-30T06:35:34.773ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b10278e2-58e0-46f8-83da-7019456ea77b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22502025-04-30T06:35:34.774ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
22512025-04-30T06:35:34.775ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22522025-04-30T06:35:34.775ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22532025-04-30T06:35:34.776ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22542025-04-30T06:35:34.779ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4c44ccd-b47f-4f99-bf72-95d4ac9a2053", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22552025-04-30T06:35:34.779ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
22562025-04-30T06:35:34.779ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22572025-04-30T06:35:34.780ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22582025-04-30T06:35:34.780ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22592025-04-30T06:35:34.780ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22602025-04-30T06:35:34.780ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22612025-04-30T06:35:34.783ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8bbcafa-ab27-41a7-980a-8a009d38e7ab", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22622025-04-30T06:35:34.783ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
22632025-04-30T06:35:34.783ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22642025-04-30T06:35:34.783ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22652025-04-30T06:35:34.783ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22662025-04-30T06:35:34.784ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22672025-04-30T06:35:34.786ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6aaba2b8-d433-4efa-bb58-d8ab13534243", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22682025-04-30T06:35:34.786ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
22692025-04-30T06:35:34.787ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22702025-04-30T06:35:34.787ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22712025-04-30T06:35:34.787ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22722025-04-30T06:35:34.787ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22732025-04-30T06:35:34.789ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
22742025-04-30T06:35:34.796ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
22752025-04-30T06:35:34.796ZTRACtest_metrics (InternalApi): successful lookup of siloed resource "VpcSubnet" using built-in user
22762025-04-30T06:35:34.796ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") }
22772025-04-30T06:35:34.796ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-4401-4000-8000-000000000000 resource_type = Project
22782025-04-30T06:35:34.796ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22792025-04-30T06:35:34.796ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22802025-04-30T06:35:34.797ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22812025-04-30T06:35:34.798ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f2f00b2-5a21-4796-88d7-f28264a937f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22822025-04-30T06:35:34.798ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
22832025-04-30T06:35:34.799ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-5110-4000-8000-000000000001 resource_type = Silo
22842025-04-30T06:35:34.799ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22852025-04-30T06:35:34.800ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22862025-04-30T06:35:34.800ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22872025-04-30T06:35:34.808ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db9851ad-badb-4e1e-bd36-34cbd03eabe0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22882025-04-30T06:35:34.808ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
22892025-04-30T06:35:34.811ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b484c172-43bf-4697-b045-c07bede71ec0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22902025-04-30T06:35:34.811ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
22912025-04-30T06:35:34.813ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
22922025-04-30T06:35:34.813ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
22932025-04-30T06:35:34.813ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
22942025-04-30T06:35:34.813ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
22952025-04-30T06:35:34.816ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61cc3db4-99cf-4857-82be-7d8ca6b2eca0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22962025-04-30T06:35:34.816ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
22972025-04-30T06:35:34.817ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fbb1880-38de-45b8-bc0f-9cff3dbdd582", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
22982025-04-30T06:35:34.817ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
22992025-04-30T06:35:34.818ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23002025-04-30T06:35:34.824ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = VpcSubnet { parent: Vpc { parent: Project { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000001, lookup_type: ById(001de000-5110-4000-8000-000000000001) }, key: 001de000-4401-4000-8000-000000000000, lookup_type: ById(001de000-4401-4000-8000-000000000000) }, key: 001de000-074c-4000-8000-000000000000, lookup_type: ByName("oxide-services") }, key: 001de000-c470-4000-8000-000000000002, lookup_type: ByName("nexus") } result = Ok(())
23012025-04-30T06:35:34.824ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23022025-04-30T06:35:34.824ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23032025-04-30T06:35:34.824ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23042025-04-30T06:35:34.826ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bf52509-c01e-4fcf-ad9f-e67cdb2000e4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23052025-04-30T06:35:34.826ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
23062025-04-30T06:35:34.829ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b421780-1501-4467-9e55-5db2fdeb55a8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23072025-04-30T06:35:34.829ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
23082025-04-30T06:35:34.829ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7531a03-2431-4517-99f2-ae091617c2b6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23092025-04-30T06:35:34.829ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
23102025-04-30T06:35:34.830ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06a68c4c-0d7b-449d-bea1-71da34a414d1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23112025-04-30T06:35:34.831ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
23122025-04-30T06:35:34.833ZDEBGtest_metrics (InternalApi): resolved names for firewall rules instance_interfaces = {} subnet_interfaces = {Name("nexus"): [NetworkInterface { id: b0e1a047-e025-4fdc-a488-6ad3731a6e2b, kind: Service { id: 588db90d-5640-4be3-9102-2ebf0af0c9a3 }, name: Name("nexus-588db90d-5640-4be3-9102-2ebf0af0c9a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }], Name("external-dns"): [NetworkInterface { id: 0671dab9-fd2e-4608-bc87-2dab839ac88f, kind: Service { id: d51ab00e-a3ad-4cf7-984a-52d44e99a71f }, name: Name("external-dns-d51ab00e-a3ad-4cf7-984a-52d44e99a71f"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }]} subnet_networks = {Name("nexus"): [V4(Ipv4Network { addr: 172.30.2.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:2::, prefix: 64 })], Name("external-dns"): [V4(Ipv4Network { addr: 172.30.1.0, prefix: 24 }), V6(Ipv6Network { addr: fd77:e9d2:9cd9:1::, prefix: 64 })]} vpc_interfaces = {}
23132025-04-30T06:35:34.833ZTRACtest_metrics (InternalApi): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
23142025-04-30T06:35:34.833ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23152025-04-30T06:35:34.833ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23162025-04-30T06:35:34.833ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23172025-04-30T06:35:34.833ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23182025-04-30T06:35:34.835ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9279e2f2-d57c-4d2d-aab3-6ec76f6a6415", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23192025-04-30T06:35:34.835ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
23202025-04-30T06:35:34.835ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23212025-04-30T06:35:34.839ZDEBGtest_metrics (InternalApi): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
23222025-04-30T06:35:34.839ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23232025-04-30T06:35:34.839ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23242025-04-30T06:35:34.839ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23252025-04-30T06:35:34.843ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d88bd470-7b7b-46b8-82ad-4fd05758f5d1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23262025-04-30T06:35:34.843ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
23272025-04-30T06:35:34.843ZTRACtest_metrics (InternalApi): fetched allowlist from DB allowed = AllowList { id: 001de000-a110-4000-8000-000000000000, time_created: 2025-04-30T06:06:08.248500Z, time_modified: 2025-04-30T06:35:33.309753Z, allowed_ips: None }
23282025-04-30T06:35:34.843ZDEBGtest_metrics (InternalApi): Allowlist for user-facing services is set to allow any inbound traffic. Existing VPC firewall rules will not be modified.
23292025-04-30T06:35:34.843ZDEBGtest_metrics (InternalApi): resolved firewall rules for sled agents sled_agent_rules = [ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("53")]), filter_protocols: Some([Udp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: 0671dab9-fd2e-4608-bc87-2dab839ac88f, kind: Service { id: d51ab00e-a3ad-4cf7-984a-52d44e99a71f }, name: Name("external-dns-d51ab00e-a3ad-4cf7-984a-52d44e99a71f"), ip: 172.30.1.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 0])), subnet: V4(Ipv4Net { addr: 172.30.1.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }, ResolvedVpcFirewallRule { action: Allow, direction: Inbound, filter_hosts: None, filter_ports: Some([L4PortRange("80"), L4PortRange("443")]), filter_protocols: Some([Tcp]), priority: 65534, status: Enabled, targets: [NetworkInterface { id: b0e1a047-e025-4fdc-a488-6ad3731a6e2b, kind: Service { id: 588db90d-5640-4be3-9102-2ebf0af0c9a3 }, name: Name("nexus-588db90d-5640-4be3-9102-2ebf0af0c9a3"), ip: 172.30.2.6, mac: MacAddr(MacAddr6([168, 64, 37, 255, 128, 1])), subnet: V4(Ipv4Net { addr: 172.30.2.0, width: 24 }), vni: Vni(100), primary: true, slot: 0, transit_ips: [] }] }]
23302025-04-30T06:35:34.843ZDEBGtest_metrics (InternalApi): resolved 2 rules for sleds
23312025-04-30T06:35:34.846ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d05d15a6-c60b-4803-9c2c-e8fdd396478e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23322025-04-30T06:35:34.846ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
23332025-04-30T06:35:34.850ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae87809d-712c-40f8-b15f-75a01713e5b4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23342025-04-30T06:35:34.850ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
23352025-04-30T06:35:34.853ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3921d367-e834-4e4a-9f2c-08c4dba0d6a0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23362025-04-30T06:35:34.853ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
23372025-04-30T06:35:34.856ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ed7ff5-8a60-49c0-8dce-0dca8361799d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23382025-04-30T06:35:34.856ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
23392025-04-30T06:35:34.857ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3c07d89-7642-4814-a709-0b456994f05f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23402025-04-30T06:35:34.857ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
23412025-04-30T06:35:34.862ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4871c503-d142-4d24-904b-dfcef7057ac7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23422025-04-30T06:35:34.862ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
23432025-04-30T06:35:34.863ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c49973eb-8d62-44a9-92a5-96d730b8ff47", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23442025-04-30T06:35:34.863ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
23452025-04-30T06:35:34.865ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e29680fe-677b-4f78-b7cb-dfbfac1c5086", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23462025-04-30T06:35:34.865ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
23472025-04-30T06:35:34.865ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23482025-04-30T06:35:34.865ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23492025-04-30T06:35:34.865ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23502025-04-30T06:35:34.865ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23512025-04-30T06:35:34.885ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23522025-04-30T06:35:34.889ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23532025-04-30T06:35:34.889ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
23542025-04-30T06:35:34.889ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23552025-04-30T06:35:34.889ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23562025-04-30T06:35:34.889ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23572025-04-30T06:35:34.890ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23582025-04-30T06:35:34.895ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23592025-04-30T06:35:34.899ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
23602025-04-30T06:35:34.900ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23612025-04-30T06:35:34.900ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23622025-04-30T06:35:34.900ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23632025-04-30T06:35:34.970ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
23642025-04-30T06:35:34.970ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
23652025-04-30T06:35:34.971ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:45048 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23662025-04-30T06:35:34.971ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:45048 req_id = b864e832-a8fc-465a-807e-717775189e02 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
23672025-04-30T06:35:34.972ZINFOtest_metrics (dropshot): request completed latency_us = 622 local_addr = [::1]:56237 method = GET remote_addr = [::1]:45048 req_id = b864e832-a8fc-465a-807e-717775189e02 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
23682025-04-30T06:35:34.972ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b864e832-a8fc-465a-807e-717775189e02", "content-length": "2769", "date": "Wed, 30 Apr 2025 06:35:33 GMT"} })
23692025-04-30T06:35:34.975ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
23702025-04-30T06:35:34.975ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
23712025-04-30T06:35:34.975ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
23722025-04-30T06:35:34.978ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
23732025-04-30T06:35:34.979ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23742025-04-30T06:35:34.979ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23752025-04-30T06:35:34.979ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23762025-04-30T06:35:34.979ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23772025-04-30T06:35:34.984ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23782025-04-30T06:35:34.989ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
23792025-04-30T06:35:34.989ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
23802025-04-30T06:35:34.990ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
23812025-04-30T06:35:34.990ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
23822025-04-30T06:35:35.018ZDEBGtest_metrics (InternalApi): resolved sleds for vpc oxide-services vpc_to_sled = [Sled { identity: SledIdentity { id: b6d65341-167c-41df-9b5c-41cded99c229, time_created: 2025-04-30T06:35:27.434010Z, time_modified: 2025-04-30T06:35:27.434010Z }, time_deleted: None, rcgen: Generation(Generation(1)), rack_id: c19a698f-c6f9-4a17-ae30-20d711b8f7dc, is_scrimlet: true, serial_number: "sim-b6d65341", part_number: "Unknown", revision: SqlU32(0), usable_hardware_threads: SqlU32(16), usable_physical_ram: ByteCount(ByteCount(34359738368)), reservoir_size: ByteCount(ByteCount(17179869184)), ip: ::1, port: SqlU16(56237), last_used_address: ::ffff, policy: InService, state: Active, sled_agent_gen: Generation(Generation(1)), repo_depot_port: SqlU16(54162) }]
23832025-04-30T06:35:35.018ZDEBGtest_metrics (InternalApi): sending firewall rules to sled agents
23842025-04-30T06:35:35.018ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23852025-04-30T06:35:35.018ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23862025-04-30T06:35:35.019ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23872025-04-30T06:35:35.023ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
23882025-04-30T06:35:35.023ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
23892025-04-30T06:35:35.024ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
23902025-04-30T06:35:35.024ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
23912025-04-30T06:35:35.024ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
23922025-04-30T06:35:35.027ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
23932025-04-30T06:35:35.032ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
23942025-04-30T06:35:35.088ZDEBGtest_metrics (InternalApi): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 body = Some(Body) method = PUT uri = http://[::1]:56237/vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23952025-04-30T06:35:35.088ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:49859 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
23962025-04-30T06:35:35.089ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:49859 req_id = 193e8de1-24a0-4946-8af6-1654635f3117 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23972025-04-30T06:35:35.090ZINFOtest_metrics (dropshot): request completed latency_us = 948 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:49859 req_id = 193e8de1-24a0-4946-8af6-1654635f3117 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc/001de000-074c-4000-8000-000000000000/firewall/rules
23982025-04-30T06:35:35.091ZDEBGtest_metrics (InternalApi): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 result = Ok(Response { url: "http://[::1]:56237/vpc/001de000-074c-4000-8000-000000000000/firewall/rules", status: 204, headers: {"x-request-id": "193e8de1-24a0-4946-8af6-1654635f3117", "date": "Wed, 30 Apr 2025 06:35:35 GMT"} })
23992025-04-30T06:35:35.093ZINFOtest_metrics (InternalApi): updated firewall rules on 1 sleds
24002025-04-30T06:35:35.093ZINFOtest_metrics (nexus): plumbed initial IP allowlist
24012025-04-30T06:35:35.162ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = / versions = all
24022025-04-30T06:35:35.162ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /access versions = all
24032025-04-30T06:35:35.162ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /assets/{path} versions = all
24042025-04-30T06:35:35.162ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /device/auth versions = all
24052025-04-30T06:35:35.162ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /device/confirm versions = all
24062025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /device/success versions = all
24072025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /device/token versions = all
24082025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /device/verify versions = all
24092025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/probes versions = all
24102025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /experimental/v1/probes versions = all
24112025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /experimental/v1/probes/{probe} versions = all
24122025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/probes/{probe} versions = all
24132025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/system/support-bundles versions = all
24142025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /experimental/v1/system/support-bundles versions = all
24152025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
24162025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
24172025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24182025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
24192025-04-30T06:35:35.163ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24202025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
24212025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
24222025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /images versions = all
24232025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /login versions = all
24242025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /login/{silo_name}/local versions = all
24252025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
24262025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
24272025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
24282025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /lookup/{path} versions = all
24292025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /projects/{path} versions = all
24302025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /projects-new versions = all
24312025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /settings/{path} versions = all
24322025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /system/{path} versions = all
24332025-04-30T06:35:35.164ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /utilization versions = all
24342025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/affinity-groups versions = all
24352025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/affinity-groups versions = all
24362025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
24372025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
24382025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
24392025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
24402025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24412025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24422025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
24432025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/anti-affinity-groups versions = all
24442025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/anti-affinity-groups versions = all
24452025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24462025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24472025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
24482025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
24492025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24502025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24512025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
24522025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/certificates versions = all
24532025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/certificates versions = all
24542025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/certificates/{certificate} versions = all
24552025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/certificates/{certificate} versions = all
24562025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/disks versions = all
24572025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/disks versions = all
24582025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/disks/{disk} versions = all
24592025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/disks/{disk} versions = all
24602025-04-30T06:35:35.165ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/disks/{disk}/bulk-write versions = all
24612025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
24622025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
24632025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/disks/{disk}/finalize versions = all
24642025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
24652025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/floating-ips versions = all
24662025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/floating-ips versions = all
24672025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
24682025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/floating-ips/{floating_ip} versions = all
24692025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
24702025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
24712025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
24722025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/groups versions = all
24732025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/groups/{group_id} versions = all
24742025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/images versions = all
24752025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/images versions = all
24762025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/images/{image} versions = all
24772025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/images/{image} versions = all
24782025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/images/{image}/demote versions = all
24792025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/images/{image}/promote versions = all
24802025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances versions = all
24812025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances versions = all
24822025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/instances/{instance} versions = all
24832025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance} versions = all
24842025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/instances/{instance} versions = all
24852025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
24862025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
24872025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/disks versions = all
24882025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances/{instance}/disks/attach versions = all
24892025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances/{instance}/disks/detach versions = all
24902025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/external-ips versions = all
24912025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24922025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
24932025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances/{instance}/reboot versions = all
24942025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/serial-console versions = all
24952025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
24962025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
24972025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances/{instance}/start versions = all
24982025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/instances/{instance}/stop versions = all
24992025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/internet-gateway-ip-addresses versions = all
25002025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/internet-gateway-ip-addresses versions = all
25012025-04-30T06:35:35.166ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
25022025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/internet-gateway-ip-pools versions = all
25032025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/internet-gateway-ip-pools versions = all
25042025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
25052025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/internet-gateways versions = all
25062025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/internet-gateways versions = all
25072025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
25082025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/internet-gateways/{gateway} versions = all
25092025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/ip-pools versions = all
25102025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/ip-pools/{pool} versions = all
25112025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/login/{silo_name}/local versions = all
25122025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/logout versions = all
25132025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/me versions = all
25142025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/me/groups versions = all
25152025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/me/ssh-keys versions = all
25162025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/me/ssh-keys versions = all
25172025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
25182025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
25192025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/metrics/{metric_name} versions = all
25202025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/network-interfaces versions = all
25212025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/network-interfaces versions = all
25222025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/network-interfaces/{interface} versions = all
25232025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/network-interfaces/{interface} versions = all
25242025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/network-interfaces/{interface} versions = all
25252025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/ping versions = all
25262025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/policy versions = all
25272025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/policy versions = all
25282025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/projects versions = all
25292025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/projects versions = all
25302025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/projects/{project} versions = all
25312025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/projects/{project} versions = all
25322025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/projects/{project} versions = all
25332025-04-30T06:35:35.167ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/projects/{project}/policy versions = all
25342025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/projects/{project}/policy versions = all
25352025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/snapshots versions = all
25362025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/snapshots versions = all
25372025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/snapshots/{snapshot} versions = all
25382025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/snapshots/{snapshot} versions = all
25392025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/disks versions = all
25402025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
25412025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
25422025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/racks versions = all
25432025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
25442025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/sleds versions = all
25452025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/hardware/sleds versions = all
25462025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
25472025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
25482025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
25492025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
25502025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
25512025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/switch-port versions = all
25522025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25532025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
25542025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
25552025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
25562025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
25572025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/switches versions = all
25582025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
25592025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/identity-providers versions = all
25602025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/identity-providers/local/users versions = all
25612025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
25622025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
25632025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/identity-providers/saml versions = all
25642025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
25652025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools versions = all
25662025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/ip-pools versions = all
25672025-04-30T06:35:35.168ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
25682025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools/{pool} versions = all
25692025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/ip-pools/{pool} versions = all
25702025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
25712025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
25722025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
25732025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
25742025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
25752025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25762025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
25772025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
25782025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools-service versions = all
25792025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/ip-pools-service/ranges versions = all
25802025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
25812025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
25822025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/metrics/{metric_name} versions = all
25832025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/address-lot versions = all
25842025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/networking/address-lot versions = all
25852025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
25862025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
25872025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/allow-list versions = all
25882025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/networking/allow-list versions = all
25892025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/networking/bfd-disable versions = all
25902025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/networking/bfd-enable versions = all
25912025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bfd-status versions = all
25922025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/networking/bgp versions = all
25932025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp versions = all
25942025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/networking/bgp versions = all
25952025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp-announce-set versions = all
25962025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
25972025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
25982025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
25992025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp-exported versions = all
26002025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp-message-history versions = all
26012025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
26022025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/bgp-status versions = all
26032025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/loopback-address versions = all
26042025-04-30T06:35:35.169ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/networking/loopback-address versions = all
26052025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
26062025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
26072025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/switch-port-settings versions = all
26082025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/networking/switch-port-settings versions = all
26092025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
26102025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/policy versions = all
26112025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/policy versions = all
26122025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/roles versions = all
26132025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/roles/{role_name} versions = all
26142025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/silo-quotas versions = all
26152025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/silos versions = all
26162025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/silos versions = all
26172025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/system/silos/{silo} versions = all
26182025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/silos/{silo} versions = all
26192025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
26202025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/silos/{silo}/policy versions = all
26212025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/silos/{silo}/policy versions = all
26222025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/silos/{silo}/quotas versions = all
26232025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
26242025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/system/timeseries/query versions = all
26252025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/timeseries/schemas versions = all
26262025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/update/repository versions = all
26272025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/update/repository/{system_version} versions = all
26282025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/update/target-release versions = all
26292025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/system/update/target-release versions = all
26302025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/users versions = all
26312025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/users/{user_id} versions = all
26322025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/users-builtin versions = all
26332025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/users-builtin/{user} versions = all
26342025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/utilization/silos versions = all
26352025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/system/utilization/silos/{silo} versions = all
26362025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/timeseries/query versions = all
26372025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/users versions = all
26382025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/utilization versions = all
26392025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-firewall-rules versions = all
26402025-04-30T06:35:35.170ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/vpc-firewall-rules versions = all
26412025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-router-routes versions = all
26422025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/vpc-router-routes versions = all
26432025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
26442025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-router-routes/{route} versions = all
26452025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/vpc-router-routes/{route} versions = all
26462025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-routers versions = all
26472025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/vpc-routers versions = all
26482025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/vpc-routers/{router} versions = all
26492025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-routers/{router} versions = all
26502025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/vpc-routers/{router} versions = all
26512025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-subnets versions = all
26522025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/vpc-subnets versions = all
26532025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
26542025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-subnets/{subnet} versions = all
26552025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
26562025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
26572025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpcs versions = all
26582025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/vpcs versions = all
26592025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/vpcs/{vpc} versions = all
26602025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/vpcs/{vpc} versions = all
26612025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/vpcs/{vpc} versions = all
26622025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/webhooks/deliveries versions = all
26632025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
26642025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/webhooks/event-classes versions = all
26652025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/webhooks/receivers versions = all
26662025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/webhooks/receivers versions = all
26672025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
26682025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
26692025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
26702025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
26712025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
26722025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
26732025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = GET path = /v1/webhooks/secrets versions = all
26742025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = POST path = /v1/webhooks/secrets versions = all
26752025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): registered endpoint local_addr = 127.0.0.1:38219 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
26762025-04-30T06:35:35.171ZINFOtest_metrics (dropshot_external): listening local_addr = 127.0.0.1:38219
26772025-04-30T06:35:35.171ZDEBGtest_metrics (dropshot_external): successfully registered DTrace USDT probes local_addr = 127.0.0.1:38219
26782025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = / versions = all
26792025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /access versions = all
26802025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /assets/{path} versions = all
26812025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /device/auth versions = all
26822025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /device/confirm versions = all
26832025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /device/success versions = all
26842025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /device/token versions = all
26852025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /device/verify versions = all
26862025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/probes versions = all
26872025-04-30T06:35:35.231ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /experimental/v1/probes versions = all
26882025-04-30T06:35:35.232ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /experimental/v1/probes/{probe} versions = all
26892025-04-30T06:35:35.232ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/probes/{probe} versions = all
26902025-04-30T06:35:35.232ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/system/support-bundles versions = all
26912025-04-30T06:35:35.232ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /experimental/v1/system/support-bundles versions = all
26922025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26932025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/system/support-bundles/{support_bundle} versions = all
26942025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26952025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download versions = all
26962025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26972025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = HEAD path = /experimental/v1/system/support-bundles/{support_bundle}/download/{file} versions = all
26982025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /experimental/v1/system/support-bundles/{support_bundle}/index versions = all
26992025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /images versions = all
27002025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /login versions = all
27012025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /login/{silo_name}/local versions = all
27022025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /login/{silo_name}/saml/{provider_name} versions = all
27032025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /login/{silo_name}/saml/{provider_name} versions = all
27042025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /login/{silo_name}/saml/{provider_name}/redirect versions = all
27052025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /lookup/{path} versions = all
27062025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /projects/{path} versions = all
27072025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /projects-new versions = all
27082025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /settings/{path} versions = all
27092025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /system/{path} versions = all
27102025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /utilization versions = all
27112025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/affinity-groups versions = all
27122025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/affinity-groups versions = all
27132025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/affinity-groups/{affinity_group} versions = all
27142025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/affinity-groups/{affinity_group} versions = all
27152025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/affinity-groups/{affinity_group} versions = all
27162025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/affinity-groups/{affinity_group}/members versions = all
27172025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27182025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27192025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/affinity-groups/{affinity_group}/members/instance/{instance} versions = all
27202025-04-30T06:35:35.233ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/anti-affinity-groups versions = all
27212025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/anti-affinity-groups versions = all
27222025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27232025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27242025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/anti-affinity-groups/{anti_affinity_group} versions = all
27252025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members versions = all
27262025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27272025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27282025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/anti-affinity-groups/{anti_affinity_group}/members/instance/{instance} versions = all
27292025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/certificates versions = all
27302025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/certificates versions = all
27312025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/certificates/{certificate} versions = all
27322025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/certificates/{certificate} versions = all
27332025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/disks versions = all
27342025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/disks versions = all
27352025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/disks/{disk} versions = all
27362025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/disks/{disk} versions = all
27372025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/disks/{disk}/bulk-write versions = all
27382025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/disks/{disk}/bulk-write-start versions = all
27392025-04-30T06:35:35.234ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/disks/{disk}/bulk-write-stop versions = all
27402025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/disks/{disk}/finalize versions = all
27412025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/disks/{disk}/metrics/{metric} versions = all
27422025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/floating-ips versions = all
27432025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/floating-ips versions = all
27442025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/floating-ips/{floating_ip} versions = all
27452025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/floating-ips/{floating_ip} versions = all
27462025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/floating-ips/{floating_ip} versions = all
27472025-04-30T06:35:35.235ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/floating-ips/{floating_ip}/attach versions = all
27482025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/floating-ips/{floating_ip}/detach versions = all
27492025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/groups versions = all
27502025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/groups/{group_id} versions = all
27512025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/images versions = all
27522025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/images versions = all
27532025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/images/{image} versions = all
27542025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/images/{image} versions = all
27552025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/images/{image}/demote versions = all
27562025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/images/{image}/promote versions = all
27572025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances versions = all
27582025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances versions = all
27592025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/instances/{instance} versions = all
27602025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance} versions = all
27612025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/instances/{instance} versions = all
27622025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/affinity-groups versions = all
27632025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/anti-affinity-groups versions = all
27642025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/disks versions = all
27652025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances/{instance}/disks/attach versions = all
27662025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances/{instance}/disks/detach versions = all
27672025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/external-ips versions = all
27682025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27692025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances/{instance}/external-ips/ephemeral versions = all
27702025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances/{instance}/reboot versions = all
27712025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/serial-console versions = all
27722025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/serial-console/stream versions = all
27732025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/instances/{instance}/ssh-public-keys versions = all
27742025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances/{instance}/start versions = all
27752025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/instances/{instance}/stop versions = all
27762025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/internet-gateway-ip-addresses versions = all
27772025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/internet-gateway-ip-addresses versions = all
27782025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/internet-gateway-ip-addresses/{address} versions = all
27792025-04-30T06:35:35.236ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/internet-gateway-ip-pools versions = all
27802025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/internet-gateway-ip-pools versions = all
27812025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/internet-gateway-ip-pools/{pool} versions = all
27822025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/internet-gateways versions = all
27832025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/internet-gateways versions = all
27842025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/internet-gateways/{gateway} versions = all
27852025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/internet-gateways/{gateway} versions = all
27862025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/ip-pools versions = all
27872025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/ip-pools/{pool} versions = all
27882025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/login/{silo_name}/local versions = all
27892025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/logout versions = all
27902025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/me versions = all
27912025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/me/groups versions = all
27922025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/me/ssh-keys versions = all
27932025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/me/ssh-keys versions = all
27942025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/me/ssh-keys/{ssh_key} versions = all
27952025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/me/ssh-keys/{ssh_key} versions = all
27962025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/metrics/{metric_name} versions = all
27972025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/network-interfaces versions = all
27982025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/network-interfaces versions = all
27992025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/network-interfaces/{interface} versions = all
28002025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/network-interfaces/{interface} versions = all
28012025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/network-interfaces/{interface} versions = all
28022025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/ping versions = all
28032025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/policy versions = all
28042025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/policy versions = all
28052025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/projects versions = all
28062025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/projects versions = all
28072025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/projects/{project} versions = all
28082025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/projects/{project} versions = all
28092025-04-30T06:35:35.237ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/projects/{project} versions = all
28102025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/projects/{project}/policy versions = all
28112025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/projects/{project}/policy versions = all
28122025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/snapshots versions = all
28132025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/snapshots versions = all
28142025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/snapshots/{snapshot} versions = all
28152025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/snapshots/{snapshot} versions = all
28162025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/disks versions = all
28172025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/disks/{disk_id} versions = all
28182025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/rack-switch-port/{rack_id}/{switch_location}/{port}/lldp/neighbors versions = all
28192025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/racks versions = all
28202025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/racks/{rack_id} versions = all
28212025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/sleds versions = all
28222025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/hardware/sleds versions = all
28232025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/sleds/{sled_id} versions = all
28242025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/sleds/{sled_id}/disks versions = all
28252025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/sleds/{sled_id}/instances versions = all
28262025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/hardware/sleds/{sled_id}/provision-policy versions = all
28272025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/sleds-uninitialized versions = all
28282025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/switch-port versions = all
28292025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28302025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/hardware/switch-port/{port}/lldp/config versions = all
28312025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/hardware/switch-port/{port}/settings versions = all
28322025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/hardware/switch-port/{port}/settings versions = all
28332025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/switch-port/{port}/status versions = all
28342025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/switches versions = all
28352025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/hardware/switches/{switch_id} versions = all
28362025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/identity-providers versions = all
28372025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/identity-providers/local/users versions = all
28382025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/identity-providers/local/users/{user_id} versions = all
28392025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/identity-providers/local/users/{user_id}/set-password versions = all
28402025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/identity-providers/saml versions = all
28412025-04-30T06:35:35.238ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/identity-providers/saml/{provider} versions = all
28422025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools versions = all
28432025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/ip-pools versions = all
28442025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/ip-pools/{pool} versions = all
28452025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools/{pool} versions = all
28462025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/ip-pools/{pool} versions = all
28472025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools/{pool}/ranges versions = all
28482025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/ip-pools/{pool}/ranges/add versions = all
28492025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/ip-pools/{pool}/ranges/remove versions = all
28502025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools/{pool}/silos versions = all
28512025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/ip-pools/{pool}/silos versions = all
28522025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28532025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/ip-pools/{pool}/silos/{silo} versions = all
28542025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools/{pool}/utilization versions = all
28552025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools-service versions = all
28562025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/ip-pools-service/ranges versions = all
28572025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/ip-pools-service/ranges/add versions = all
28582025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/ip-pools-service/ranges/remove versions = all
28592025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/metrics/{metric_name} versions = all
28602025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/address-lot versions = all
28612025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/networking/address-lot versions = all
28622025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/networking/address-lot/{address_lot} versions = all
28632025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/address-lot/{address_lot}/blocks versions = all
28642025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/allow-list versions = all
28652025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/networking/allow-list versions = all
28662025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/networking/bfd-disable versions = all
28672025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/networking/bfd-enable versions = all
28682025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bfd-status versions = all
28692025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/networking/bgp versions = all
28702025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp versions = all
28712025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/networking/bgp versions = all
28722025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp-announce-set versions = all
28732025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/networking/bgp-announce-set versions = all
28742025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/networking/bgp-announce-set/{announce_set} versions = all
28752025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp-announce-set/{announce_set}/announcement versions = all
28762025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp-exported versions = all
28772025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp-message-history versions = all
28782025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp-routes-ipv4 versions = all
28792025-04-30T06:35:35.239ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/bgp-status versions = all
28802025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/loopback-address versions = all
28812025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/networking/loopback-address versions = all
28822025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/networking/loopback-address/{rack_id}/{switch_location}/{address}/{subnet_mask} versions = all
28832025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/networking/switch-port-settings versions = all
28842025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/switch-port-settings versions = all
28852025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/networking/switch-port-settings versions = all
28862025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/networking/switch-port-settings/{port} versions = all
28872025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/policy versions = all
28882025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/policy versions = all
28892025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/roles versions = all
28902025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/roles/{role_name} versions = all
28912025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/silo-quotas versions = all
28922025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/silos versions = all
28932025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/silos versions = all
28942025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/system/silos/{silo} versions = all
28952025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/silos/{silo} versions = all
28962025-04-30T06:35:35.240ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/silos/{silo}/ip-pools versions = all
28972025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/silos/{silo}/policy versions = all
28982025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/silos/{silo}/policy versions = all
28992025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/silos/{silo}/quotas versions = all
29002025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/silos/{silo}/quotas versions = all
29012025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/system/timeseries/query versions = all
29022025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/timeseries/schemas versions = all
29032025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/update/repository versions = all
29042025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/update/repository/{system_version} versions = all
29052025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/update/target-release versions = all
29062025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/system/update/target-release versions = all
29072025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/users versions = all
29082025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/users/{user_id} versions = all
29092025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/users-builtin versions = all
29102025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/users-builtin/{user} versions = all
29112025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/utilization/silos versions = all
29122025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/system/utilization/silos/{silo} versions = all
29132025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/timeseries/query versions = all
29142025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/users versions = all
29152025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/utilization versions = all
29162025-04-30T06:35:35.241ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-firewall-rules versions = all
29172025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/vpc-firewall-rules versions = all
29182025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-router-routes versions = all
29192025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/vpc-router-routes versions = all
29202025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/vpc-router-routes/{route} versions = all
29212025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-router-routes/{route} versions = all
29222025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/vpc-router-routes/{route} versions = all
29232025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-routers versions = all
29242025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/vpc-routers versions = all
29252025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/vpc-routers/{router} versions = all
29262025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-routers/{router} versions = all
29272025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/vpc-routers/{router} versions = all
29282025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-subnets versions = all
29292025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/vpc-subnets versions = all
29302025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/vpc-subnets/{subnet} versions = all
29312025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-subnets/{subnet} versions = all
29322025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/vpc-subnets/{subnet} versions = all
29332025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpc-subnets/{subnet}/network-interfaces versions = all
29342025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpcs versions = all
29352025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/vpcs versions = all
29362025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/vpcs/{vpc} versions = all
29372025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/vpcs/{vpc} versions = all
29382025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/vpcs/{vpc} versions = all
29392025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/webhooks/deliveries versions = all
29402025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/webhooks/deliveries/{event_id}/resend versions = all
29412025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/webhooks/event-classes versions = all
29422025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/webhooks/receivers versions = all
29432025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/webhooks/receivers versions = all
29442025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/webhooks/receivers/{receiver} versions = all
29452025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/webhooks/receivers/{receiver} versions = all
29462025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = PUT path = /v1/webhooks/receivers/{receiver} versions = all
29472025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/webhooks/receivers/{receiver}/probe versions = all
29482025-04-30T06:35:35.242ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/webhooks/receivers/{receiver}/subscriptions versions = all
29492025-04-30T06:35:35.243ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/webhooks/receivers/{receiver}/subscriptions/{subscription} versions = all
29502025-04-30T06:35:35.243ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = GET path = /v1/webhooks/secrets versions = all
29512025-04-30T06:35:35.243ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = POST path = /v1/webhooks/secrets versions = all
29522025-04-30T06:35:35.243ZDEBGtest_metrics (dropshot_external_techport): registered endpoint local_addr = 127.0.0.1:49059 method = DELETE path = /v1/webhooks/secrets/{secret_id} versions = all
29532025-04-30T06:35:35.243ZINFOtest_metrics (dropshot_external_techport): listening local_addr = 127.0.0.1:49059
29542025-04-30T06:35:35.243ZDEBGtest_metrics (dropshot_external_techport): successfully registered DTrace USDT probes local_addr = 127.0.0.1:49059
29552025-04-30T06:35:35.253ZDEBGtest_metrics (nexus-producer-server): registered DTrace probes
29562025-04-30T06:35:35.254ZDEBGtest_metrics (dropshot): registered endpoint local_addr = 127.0.0.1:65131 method = GET path = /{producer_id} versions = all
29572025-04-30T06:35:35.254ZINFOtest_metrics (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = 127.0.0.1:65131
29582025-04-30T06:35:35.255ZDEBGtest_metrics (dropshot): successfully registered DTrace USDT probes local_addr = 127.0.0.1:65131
29592025-04-30T06:35:35.255ZDEBGtest_metrics (nexus-producer-server): Requested any available port, Dropshot server has been bound to 127.0.0.1:65131
29602025-04-30T06:35:35.255ZDEBGtest_metrics (nexus-producer-server): Nexus IP provided explicitly, will use it addr = 127.0.0.1:49095
29612025-04-30T06:35:35.255ZDEBGtest_metrics (nexus-producer-server): starting producer registration task
29622025-04-30T06:35:35.256ZINFOtest_metrics (nexus-producer-server): starting oximeter metric producer server address = 127.0.0.1:65131 file = oximeter/producer/src/lib.rs:279 interval = 10s producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
29632025-04-30T06:35:35.256ZDEBGtest_metrics (init_with_steps): Running step start_oximeter
29642025-04-30T06:35:35.256ZDEBGtest_metrics: Starting Oximeter
29652025-04-30T06:35:35.259ZDEBGtest_metrics (oximeter): registered DTrace probes
29662025-04-30T06:35:35.259ZINFOtest_metrics (oximeter): starting oximeter server file = oximeter/collector/src/lib.rs:236
29672025-04-30T06:35:35.259ZDEBGtest_metrics (oximeter): creating ClickHouse client
29682025-04-30T06:35:35.260ZDEBGtest_metrics (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
29692025-04-30T06:35:35.261ZDEBGtest_metrics (producer-registration-task): registering as oximeter producer with Nexus registration_count = 0
29702025-04-30T06:35:35.261ZDEBGtest_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:49095
29712025-04-30T06:35:35.373ZDEBGtest_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:49095/metrics/producers
29722025-04-30T06:35:35.374ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:36616
29732025-04-30T06:35:35.404ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36616 req_id = 100ba6ad-0219-4c39-b857-58302342f0d8 uri = /metrics/producers
29742025-04-30T06:35:35.407ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36616 req_id = 100ba6ad-0219-4c39-b857-58302342f0d8 resource = Database uri = /metrics/producers
29752025-04-30T06:35:35.407ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36616 req_id = 100ba6ad-0219-4c39-b857-58302342f0d8 roles = RoleSet { roles: {} } uri = /metrics/producers
29762025-04-30T06:35:35.408ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36616 req_id = 100ba6ad-0219-4c39-b857-58302342f0d8 resource = Database result = Ok(()) uri = /metrics/producers
29772025-04-30T06:35:35.412ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29782025-04-30T06:35:35.429ZTRACtest_metrics (dns): buffer buffer = Length: 79 (0x4f) bytes\n0000: 6c ca 01 00 00 01 00 00 00 00 00 01 10 5f 63 72 l............_cr\n0010: 75 63 69 62 6c 65 2d 70 61 6e 74 72 79 04 5f 74 ucible-pantry._t\n0020: 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 cp.control-plane\n0030: 05 6f 78 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 .oxide.internal.\n0040: 00 21 00 01 00 00 29 04 d0 00 00 00 00 00 00 .!....)........ kind = dns peer_addr = [::1]:51405 req_id = a678a0b4-a29d-43e6-b8a5-c2f2c3addb56
29792025-04-30T06:35:35.429ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27850,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 16,\n 95,\n 99,\n 114,\n 117,\n 99,\n 105,\n 98,\n 108,\n 101,\n 45,\n 112,\n 97,\n 110,\n 116,\n 114,\n 121,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51405 req_id = a678a0b4-a29d-43e6-b8a5-c2f2c3addb56
29802025-04-30T06:35:35.429ZDEBGtest_metrics (store): query key key = _crucible-pantry._tcp
29812025-04-30T06:35:35.430ZDEBGtest_metrics (store): query key key = 8abe35ca-5518-49a6-8363-901ebe0a98a8.host
29822025-04-30T06:35:35.430ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51405 query = LowerQuery { name: LowerName(Name("_crucible-pantry._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_crucible-pantry._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 38907, target: Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal") })) }] req_id = a678a0b4-a29d-43e6-b8a5-c2f2c3addb56
29832025-04-30T06:35:35.431ZWARNtest_metrics (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
29842025-04-30T06:35:35.432ZDEBGtest_metrics (oximeter-agent): oximeter database does not exist, creating collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
29852025-04-30T06:35:35.432ZINFOtest_metrics (clickhouse-client): reading db version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:794 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
29862025-04-30T06:35:35.438ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
29872025-04-30T06:35:35.439ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
29882025-04-30T06:35:35.439ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.448986761s iteration = 3 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
29892025-04-30T06:35:35.439ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT MAX(value) AS max_version FROM oximeter.version;
29902025-04-30T06:35:35.443ZWARNtest_metrics (clickhouse-client): oximeter database does not exist, or is out-of-date collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:894 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
29912025-04-30T06:35:35.443ZINFOtest_metrics (clickhouse-client): read oximeter database version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:799 id = 758cdc67-47ef-43f0-b1af-382db611bb3e version = 0
29922025-04-30T06:35:35.444ZINFOtest_metrics (clickhouse-client): initializing oximeter schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:804 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
29932025-04-30T06:35:35.444ZDEBGtest_metrics (clickhouse-client): initializing ClickHouse database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
29942025-04-30T06:35:35.444ZTRACtest_metrics (dns): buffer buffer = Length: 99 (0x63) bytes\n0000: cb 2b 01 00 00 01 00 00 00 00 00 01 24 38 61 62 .+..........$8ab\n0010: 65 33 35 63 61 2d 35 35 31 38 2d 34 39 61 36 2d e35ca-5518-49a6-\n0020: 38 33 36 33 2d 39 30 31 65 62 65 30 61 39 38 61 8363-901ebe0a98a\n0030: 38 04 68 6f 73 74 0d 63 6f 6e 74 72 6f 6c 2d 70 8.host.control-p\n0040: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0050: 6e 61 6c 00 00 1c 00 01 00 00 29 04 d0 00 00 00 nal.......).....\n0060: 00 00 00 ... kind = dns peer_addr = [::1]:49423 req_id = fdcb2ab2-850f-4185-8de6-b98137ea148e
29952025-04-30T06:35:35.444ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52011,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 36,\n 56,\n 97,\n 98,\n 101,\n 51,\n 53,\n 99,\n 97,\n 45,\n 53,\n 53,\n 49,\n 56,\n 45,\n 52,\n 57,\n 97,\n 54,\n 45,\n 56,\n 51,\n 54,\n 51,\n 45,\n 57,\n 48,\n 49,\n 101,\n 98,\n 101,\n 48,\n 97,\n 57,\n 56,\n 97,\n 56,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49423 req_id = fdcb2ab2-850f-4185-8de6-b98137ea148e
29962025-04-30T06:35:35.444ZDEBGtest_metrics (store): query key key = 8abe35ca-5518-49a6-8363-901ebe0a98a8.host
29972025-04-30T06:35:35.444ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49423 query = LowerQuery { name: LowerName(Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal.")), original: Query { name: Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("8abe35ca-5518-49a6-8363-901ebe0a98a8.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fdcb2ab2-850f-4185-8de6-b98137ea148e
29982025-04-30T06:35:35.444ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = CREATE DATABASE IF NOT EXISTS oximeter
29992025-04-30T06:35:35.533ZINFOtest_metrics (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 128631 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36616 req_id = 100ba6ad-0219-4c39-b857-58302342f0d8 response_code = 503 uri = /metrics/producers
30002025-04-30T06:35:35.533ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\n/* The version table contains metadata about the `oximeter` database */\nCREATE TABLE IF NOT EXISTS oximeter.version\n(\n value UInt64,\n timestamp DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (value, timestamp)
30012025-04-30T06:35:35.537ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:46873 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30022025-04-30T06:35:35.538ZDEBGtest_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "100ba6ad-0219-4c39-b857-58302342f0d8", "content-length": "133", "date": "Wed, 30 Apr 2025 06:35:35 GMT"} })
30032025-04-30T06:35:35.539ZWARNtest_metrics (producer-registration-task): failed to register with Nexus, will retry count = 0 delay = 278.045217ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"100ba6ad-0219-4c39-b857-58302342f0d8\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:35:35 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"100ba6ad-0219-4c39-b857-58302342f0d8\\" }" file = oximeter/producer/src/lib.rs:371
30042025-04-30T06:35:35.539ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\n/* The measurement tables contain all individual samples from each timeseries.\n *\n * Each table stores a single datum type, and otherwise contains nearly the same\n * structure. The primary sorting key is on the timeseries name, key, and then\n * timestamp, so that all timeseries from the same schema are grouped, followed\n * by all samples from the same timeseries.\n *\n * This reflects that one usually looks up the _key_ in one or more field table,\n * and then uses that to index quickly into the measurements tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Bool)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30052025-04-30T06:35:35.541ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:46873 req_id = 5c0f877e-6135-46c5-a5c1-6c624388aa3a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30062025-04-30T06:35:35.543ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2122 local_addr = [::1]:38907 method = GET remote_addr = [::1]:46873 req_id = 5c0f877e-6135-46c5-a5c1-6c624388aa3a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30072025-04-30T06:35:35.543ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30082025-04-30T06:35:35.972ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30092025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:64485 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30102025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:64250 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30112025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:63543 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30122025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:33318 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30132025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:48452 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30142025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:42219 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30152025-04-30T06:35:35.972ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:62630 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
30162025-04-30T06:35:35.973ZDEBGtest_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:49095
30172025-04-30T06:35:36.067ZDEBGtest_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:49095/metrics/producers
30182025-04-30T06:35:36.074ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30192025-04-30T06:35:36.075ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:47617
30202025-04-30T06:35:36.075ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:64485 req_id = 6ee80b11-824c-40ce-8482-0460c1333fbe sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30212025-04-30T06:35:36.075ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:64250 req_id = e69772e0-95b0-423d-a2f0-dd55070dba4b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30222025-04-30T06:35:36.076ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:63543 req_id = 0d454546-ffe7-405d-9ee4-4057d97ae147 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30232025-04-30T06:35:36.076ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:33318 req_id = ee8be317-29f5-4751-a839-12df67232ddb sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30242025-04-30T06:35:36.076ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:48452 req_id = acb214b8-87b4-4f18-be9a-0d08b96ba08e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30252025-04-30T06:35:36.076ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:42219 req_id = c3fe879d-cfcf-4cdf-bd9c-390fe0a65502 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30262025-04-30T06:35:36.076ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:62630 req_id = 80d89b31-a3db-4200-af2f-cb4ba87abadd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30272025-04-30T06:35:36.077ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2202 local_addr = [::1]:38907 method = GET remote_addr = [::1]:64485 req_id = 6ee80b11-824c-40ce-8482-0460c1333fbe response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30282025-04-30T06:35:36.078ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2517 local_addr = [::1]:38907 method = GET remote_addr = [::1]:64250 req_id = e69772e0-95b0-423d-a2f0-dd55070dba4b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30292025-04-30T06:35:36.078ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2339 local_addr = [::1]:38907 method = GET remote_addr = [::1]:63543 req_id = 0d454546-ffe7-405d-9ee4-4057d97ae147 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30302025-04-30T06:35:36.078ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2250 local_addr = [::1]:38907 method = GET remote_addr = [::1]:33318 req_id = ee8be317-29f5-4751-a839-12df67232ddb response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30312025-04-30T06:35:36.078ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2204 local_addr = [::1]:38907 method = GET remote_addr = [::1]:48452 req_id = acb214b8-87b4-4f18-be9a-0d08b96ba08e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30322025-04-30T06:35:36.078ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2162 local_addr = [::1]:38907 method = GET remote_addr = [::1]:42219 req_id = c3fe879d-cfcf-4cdf-bd9c-390fe0a65502 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30332025-04-30T06:35:36.079ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 2066 local_addr = [::1]:38907 method = GET remote_addr = [::1]:62630 req_id = 80d89b31-a3db-4200-af2f-cb4ba87abadd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
30342025-04-30T06:35:36.080ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt16)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30352025-04-30T06:35:36.084ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:47617 req_id = 8b71bb60-59f6-499f-9f4e-9e575d4d7980 uri = /metrics/producers
30362025-04-30T06:35:36.088ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:47617 req_id = 8b71bb60-59f6-499f-9f4e-9e575d4d7980 resource = Database uri = /metrics/producers
30372025-04-30T06:35:36.088ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:47617 req_id = 8b71bb60-59f6-499f-9f4e-9e575d4d7980 roles = RoleSet { roles: {} } uri = /metrics/producers
30382025-04-30T06:35:36.089ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:47617 req_id = 8b71bb60-59f6-499f-9f4e-9e575d4d7980 resource = Database result = Ok(()) uri = /metrics/producers
30392025-04-30T06:35:36.090ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30402025-04-30T06:35:36.153ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30412025-04-30T06:35:36.166ZINFOtest_metrics (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 81290 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:47617 req_id = 8b71bb60-59f6-499f-9f4e-9e575d4d7980 response_code = 503 uri = /metrics/producers
30422025-04-30T06:35:36.166ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30432025-04-30T06:35:36.172ZDEBGtest_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "8b71bb60-59f6-499f-9f4e-9e575d4d7980", "content-length": "133", "date": "Wed, 30 Apr 2025 06:35:36 GMT"} })
30442025-04-30T06:35:36.174ZWARNtest_metrics (producer-registration-task): failed to register with Nexus, will retry count = 1 delay = 912.850945ms error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"8b71bb60-59f6-499f-9f4e-9e575d4d7980\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:35:36 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"8b71bb60-59f6-499f-9f4e-9e575d4d7980\\" }" file = oximeter/producer/src/lib.rs:371
30452025-04-30T06:35:36.174ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30462025-04-30T06:35:36.199ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f32\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30472025-04-30T06:35:36.203ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_f64\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30482025-04-30T06:35:36.206ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(String)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30492025-04-30T06:35:36.209ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_bytes\n(\n timeseries_name String,\n timeseries_key UInt64,\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Right now we can't unambiguously record a nullable byte array.\n * Arrays cannot be nested in `Nullable()` types, and encoding the array as\n * a string isn't palatable for a few reasons.\n * See: https://github.com/oxidecomputer/omicron/issues/4551 for more\n * details.\n */\n datum Array(UInt8)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30502025-04-30T06:35:36.212ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativei64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Int64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30512025-04-30T06:35:36.215ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativeu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(UInt64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30522025-04-30T06:35:36.218ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float32)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30532025-04-30T06:35:36.220ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_cumulativef64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n datum Nullable(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30542025-04-30T06:35:36.223ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n /*\n * NOTE: Array types cannot be Nullable, see\n * https://clickhouse.com/docs/en/sql-reference/data-types/nullable\n * for more details.\n *\n * This means we need to use empty arrays to indicate a missing value. This\n * is unfortunate, and at this point relies on the fact that an\n * `oximeter::Histogram` cannot have zero bins. If that changes, we'll need\n * to figure out another way to represent missing samples here.\n */\n bins Array(Int8),\n counts Array(UInt64),\n min Int8,\n max Int8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30552025-04-30T06:35:36.226ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu8\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt8),\n counts Array(UInt64),\n min UInt8,\n max UInt8,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30562025-04-30T06:35:36.230ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int16),\n counts Array(UInt64),\n min Int16,\n max Int16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30572025-04-30T06:35:36.233ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu16\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt16),\n counts Array(UInt64),\n min UInt16,\n max UInt16,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30582025-04-30T06:35:36.238ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int32),\n counts Array(UInt64),\n min Int32,\n max Int32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30592025-04-30T06:35:36.248ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt32),\n counts Array(UInt64),\n min UInt32,\n max UInt32,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30602025-04-30T06:35:36.253ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogrami64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Int64),\n counts Array(UInt64),\n min Int64,\n max Int64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30612025-04-30T06:35:36.271ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramu64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(UInt64),\n counts Array(UInt64),\n min UInt64,\n max UInt64,\n sum_of_samples Int64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30622025-04-30T06:35:36.276ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf32\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float32),\n counts Array(UInt64),\n min Float32,\n max Float32,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30632025-04-30T06:35:36.287ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.measurements_histogramf64\n(\n timeseries_name String,\n timeseries_key UInt64,\n start_time DateTime64(9, 'UTC'),\n timestamp DateTime64(9, 'UTC'),\n bins Array(Float64),\n counts Array(UInt64),\n min Float64,\n max Float64,\n sum_of_samples Float64,\n squared_mean Float64,\n p50_marker_heights Array(Float64),\n p50_marker_positions Array(UInt64),\n p50_desired_marker_positions Array(Float64),\n p90_marker_heights Array(Float64),\n p90_marker_positions Array(UInt64),\n p90_desired_marker_positions Array(Float64),\n p99_marker_heights Array(Float64),\n p99_marker_positions Array(UInt64),\n p99_desired_marker_positions Array(Float64)\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, timeseries_key, start_time, timestamp)\nTTL toDateTime(timestamp) + INTERVAL 30 DAY
30642025-04-30T06:35:36.297ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\n/* The field tables store named dimensions of each timeseries.\n *\n * As with the measurement tables, there is one field table for each field data\n * type. Fields are deduplicated by using the "replacing merge tree", though\n * this behavior **must not** be relied upon for query correctness.\n *\n * The index for the fields differs from the measurements, however. Rows are\n * sorted by timeseries name, then field name, field value, and finally\n * timeseries key. This reflects the most common pattern for looking them up:\n * by field name and possibly value, within a timeseries. The resulting keys are\n * usually then used to look up measurements.\n *\n * NOTE: We may want to consider a secondary index on these tables, sorting by\n * timeseries name and then key, since it would improve lookups where one\n * already has the key. Realistically though, these tables are quite small and\n * so performance benefits will be low in absolute terms.\n *\n * TTL: We use a materialized column to expire old field table records. This\n * column is generated automatically by the database whenever a new row is\n * inserted. It cannot be inserted directly, nor is it returned in a `SELECT *`\n * query. Since these tables are `ReplacingMergeTree`s, that means the last\n * record will remain during a deduplication, which will have the last\n * timestamp. ClickHouse will then expire old data for us, similar to the\n * measurement tables.\n */\nCREATE TABLE IF NOT EXISTS oximeter.fields_bool\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Bool,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30652025-04-30T06:35:36.302ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30662025-04-30T06:35:36.307ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u8\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt8,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30672025-04-30T06:35:36.358ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30682025-04-30T06:35:36.381ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u16\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt16,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30692025-04-30T06:35:36.390ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30702025-04-30T06:35:36.403ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u32\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt32,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30712025-04-30T06:35:36.407ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_i64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value Int64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30722025-04-30T06:35:36.450ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_u64\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UInt64,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30732025-04-30T06:35:36.477ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_ipaddr\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value IPv6,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30742025-04-30T06:35:36.485ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_string\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value String,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30752025-04-30T06:35:36.507ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\nCREATE TABLE IF NOT EXISTS oximeter.fields_uuid\n(\n timeseries_name String,\n timeseries_key UInt64,\n field_name String,\n field_value UUID,\n last_updated_at DateTime MATERIALIZED now()\n)\nENGINE = ReplacingMergeTree()\nORDER BY (timeseries_name, field_name, field_value, timeseries_key)\nTTL last_updated_at + INTERVAL 30 DAY
30762025-04-30T06:35:36.520ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = \n\n/* The timeseries schema table stores the extracted schema for the samples\n * oximeter collects.\n */\nCREATE TABLE IF NOT EXISTS oximeter.timeseries_schema\n(\n timeseries_name String,\n fields Nested(\n name String,\n type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'IpAddr' = 3,\n 'String' = 4,\n 'Uuid' = 6,\n 'I8' = 7,\n 'U8' = 8,\n 'I16' = 9,\n 'U16' = 10,\n 'I32' = 11,\n 'U32' = 12,\n 'U64' = 13\n ),\n source Enum(\n 'Target' = 1,\n 'Metric' = 2\n )\n ),\n datum_type Enum(\n 'Bool' = 1,\n 'I64' = 2,\n 'F64' = 3,\n 'String' = 4,\n 'Bytes' = 5,\n 'CumulativeI64' = 6,\n 'CumulativeF64' = 7,\n 'HistogramI64' = 8,\n 'HistogramF64' = 9,\n 'I8' = 10,\n 'U8' = 11,\n 'I16' = 12,\n 'U16' = 13,\n 'I32' = 14,\n 'U32' = 15,\n 'U64' = 16,\n 'F32' = 17,\n 'CumulativeU64' = 18,\n 'CumulativeF32' = 19,\n 'HistogramI8' = 20,\n 'HistogramU8' = 21,\n 'HistogramI16' = 22,\n 'HistogramU16' = 23,\n 'HistogramI32' = 24,\n 'HistogramU32' = 25,\n 'HistogramU64' = 26,\n 'HistogramF32' = 27\n ),\n created DateTime64(9, 'UTC')\n)\nENGINE = MergeTree()\nORDER BY (timeseries_name, fields.name)
30772025-04-30T06:35:36.546ZINFOtest_metrics (clickhouse-client): inserting current version collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/db/src/client/mod.rs:825 id = 758cdc67-47ef-43f0-b1af-382db611bb3e version = 14
30782025-04-30T06:35:36.546ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = INSERT INTO oximeter.version (*) VALUES (14, now());
30792025-04-30T06:35:36.559ZDEBGtest_metrics (clickhouse-client): registered USDT probes collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 4b2dbfd5-fb4f-4114-a587-22a30be63d4a
30802025-04-30T06:35:36.560ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:48893 method = GET path = /info versions = all
30812025-04-30T06:35:36.560ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:48893 method = GET path = /producers versions = all
30822025-04-30T06:35:36.560ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:48893 method = DELETE path = /producers/{producer_id} versions = all
30832025-04-30T06:35:36.560ZDEBGtest_metrics (dropshot): registered endpoint local_addr = [::1]:48893 method = GET path = /producers/{producer_id} versions = all
30842025-04-30T06:35:36.560ZINFOtest_metrics (dropshot): listening file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:328 local_addr = [::1]:48893
30852025-04-30T06:35:36.561ZDEBGtest_metrics (dropshot): successfully registered DTrace USDT probes local_addr = [::1]:48893
30862025-04-30T06:35:36.562ZDEBGtest_metrics (oximeter): registered USDT probes
30872025-04-30T06:35:36.562ZDEBGtest_metrics (oximeter): contacting nexus
30882025-04-30T06:35:37.171ZDEBGtest_metrics (oximeter): client request body = Some(Body) method = POST uri = http://127.0.0.1:49095/metrics/collectors
30892025-04-30T06:35:37.172ZDEBGtest_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:49095
30902025-04-30T06:35:37.263ZDEBGtest_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:49095/metrics/producers
30912025-04-30T06:35:37.264ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:36706
30922025-04-30T06:35:37.264ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:56586
30932025-04-30T06:35:37.265ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:56586 req_id = 347831d3-1e7b-4ef5-a359-f3b65c071939 uri = /metrics/producers
30942025-04-30T06:35:37.266ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:56586 req_id = 347831d3-1e7b-4ef5-a359-f3b65c071939 resource = Database uri = /metrics/producers
30952025-04-30T06:35:37.266ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:56586 req_id = 347831d3-1e7b-4ef5-a359-f3b65c071939 roles = RoleSet { roles: {} } uri = /metrics/producers
30962025-04-30T06:35:37.267ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:56586 req_id = 347831d3-1e7b-4ef5-a359-f3b65c071939 resource = Database result = Ok(()) uri = /metrics/producers
30972025-04-30T06:35:37.268ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36706 req_id = 64241865-4a34-45cb-b7e5-fc07a63099c4 uri = /metrics/collectors
30982025-04-30T06:35:37.268ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36706 req_id = 64241865-4a34-45cb-b7e5-fc07a63099c4 resource = Database uri = /metrics/collectors
30992025-04-30T06:35:37.268ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36706 req_id = 64241865-4a34-45cb-b7e5-fc07a63099c4 roles = RoleSet { roles: {} } uri = /metrics/collectors
31002025-04-30T06:35:37.269ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36706 req_id = 64241865-4a34-45cb-b7e5-fc07a63099c4 resource = Database result = Ok(()) uri = /metrics/collectors
31012025-04-30T06:35:37.275ZINFOtest_metrics (nexus): registered new oximeter metric collection server address = [::1]:48893 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31022025-04-30T06:35:37.276ZINFOtest_metrics (dropshot_internal): request completed error_message_external = Service Unavailable error_message_internal = no Oximeter instances available for assignment latency_us = 10818 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:56586 req_id = 347831d3-1e7b-4ef5-a359-f3b65c071939 response_code = 503 uri = /metrics/producers
31032025-04-30T06:35:37.276ZINFOtest_metrics (dropshot_internal): request completed latency_us = 8075 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:36706 req_id = 64241865-4a34-45cb-b7e5-fc07a63099c4 response_code = 204 uri = /metrics/collectors
31042025-04-30T06:35:37.278ZDEBGtest_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/producers", status: 503, headers: {"content-type": "application/json", "x-request-id": "347831d3-1e7b-4ef5-a359-f3b65c071939", "content-length": "133", "date": "Wed, 30 Apr 2025 06:35:37 GMT"} })
31052025-04-30T06:35:37.280ZWARNtest_metrics (producer-registration-task): failed to register with Nexus, will retry count = 2 delay = 2.019088071s error = "Error Response: status: 503 Service Unavailable; headers: {\\"content-type\\": \\"application/json\\", \\"x-request-id\\": \\"347831d3-1e7b-4ef5-a359-f3b65c071939\\", \\"content-length\\": \\"133\\", \\"date\\": \\"Wed, 30 Apr 2025 06:35:37 GMT\\"}; value: Error { error_code: Some(\\"ServiceNotAvailable\\"), message: \\"Service Unavailable\\", request_id: \\"347831d3-1e7b-4ef5-a359-f3b65c071939\\" }" file = oximeter/producer/src/lib.rs:371
31062025-04-30T06:35:37.280ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors", status: 204, headers: {"x-request-id": "64241865-4a34-45cb-b7e5-fc07a63099c4", "date": "Wed, 30 Apr 2025 06:35:37 GMT"} })
31072025-04-30T06:35:37.281ZINFOtest_metrics (oximeter): oximeter registered with nexus file = oximeter/collector/src/lib.rs:392 id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78
31082025-04-30T06:35:37.281ZDEBGtest_metrics (init_with_steps): Running step start_producer_server
31092025-04-30T06:35:37.281ZDEBGtest_metrics: Starting test metric Producer Server
31102025-04-30T06:35:37.293ZINFOtest_metrics (external client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:38219/v1/system/ip-pools
31112025-04-30T06:35:37.294ZINFOtest_metrics (oximeter-agent): starting refresh list task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:559
31122025-04-30T06:35:37.359ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:39396
31132025-04-30T06:35:37.359ZINFOtest_metrics (dropshot_external): accepted connection local_addr = 127.0.0.1:38219 remote_addr = 127.0.0.1:64657
31142025-04-30T06:35:37.360ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
31152025-04-30T06:35:37.361ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31162025-04-30T06:35:37.362ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 584ec3f2-795c-4b3f-8010-5dca0cdfc27f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31172025-04-30T06:35:37.362ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:39396 req_id = 98fcd458-5770-4f9b-a36a-23238e7accef uri = /metrics/producers
31182025-04-30T06:35:37.363ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 584ec3f2-795c-4b3f-8010-5dca0cdfc27f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31192025-04-30T06:35:37.363ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 584ec3f2-795c-4b3f-8010-5dca0cdfc27f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31202025-04-30T06:35:37.364ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 584ec3f2-795c-4b3f-8010-5dca0cdfc27f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31212025-04-30T06:35:37.364ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:39396 req_id = 98fcd458-5770-4f9b-a36a-23238e7accef resource = Database uri = /metrics/producers
31222025-04-30T06:35:37.365ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:39396 req_id = 98fcd458-5770-4f9b-a36a-23238e7accef roles = RoleSet { roles: {} } uri = /metrics/producers
31232025-04-30T06:35:37.365ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:39396 req_id = 98fcd458-5770-4f9b-a36a-23238e7accef resource = Database result = Ok(()) uri = /metrics/producers
31242025-04-30T06:35:37.367ZTRACtest_metrics (dropshot_external): incoming request local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 uri = /v1/system/ip-pools
31252025-04-30T06:35:37.368ZTRACtest_metrics (dropshot_external): authn: trying SchemeName("spoof") local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 uri = /v1/system/ip-pools
31262025-04-30T06:35:37.368ZDEBGtest_metrics (dropshot_external): looking up silo for user local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/system/ip-pools
31272025-04-30T06:35:37.368ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31282025-04-30T06:35:37.369ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31292025-04-30T06:35:37.373ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31302025-04-30T06:35:37.379ZINFOtest_metrics (dropshot_internal): request completed latency_us = 17201 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 584ec3f2-795c-4b3f-8010-5dca0cdfc27f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
31312025-04-30T06:35:37.387ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "584ec3f2-795c-4b3f-8010-5dca0cdfc27f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:37 GMT"} })
31322025-04-30T06:35:37.387ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 0 n_pruned_tasks = 0
31332025-04-30T06:35:37.390ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31342025-04-30T06:35:37.390ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31352025-04-30T06:35:37.391ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31362025-04-30T06:35:37.397ZTRACtest_metrics (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007, lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
31372025-04-30T06:35:37.397ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
31382025-04-30T06:35:37.397ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31392025-04-30T06:35:37.397ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31402025-04-30T06:35:37.398ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31412025-04-30T06:35:37.402ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31422025-04-30T06:35:37.402ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31432025-04-30T06:35:37.402ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31442025-04-30T06:35:37.405ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31452025-04-30T06:35:37.428ZINFOtest_metrics (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
31462025-04-30T06:35:37.429ZINFOtest_metrics (dropshot_internal): request completed latency_us = 66605 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:39396 req_id = 98fcd458-5770-4f9b-a36a-23238e7accef response_code = 201 uri = /metrics/producers
31472025-04-30T06:35:37.431ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
31482025-04-30T06:35:37.438ZDEBGtest_metrics (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007, lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
31492025-04-30T06:35:37.438ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31502025-04-30T06:35:37.438ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31512025-04-30T06:35:37.439ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31522025-04-30T06:35:37.448ZTRACtest_metrics (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
31532025-04-30T06:35:37.448ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
31542025-04-30T06:35:37.448ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31552025-04-30T06:35:37.448ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31562025-04-30T06:35:37.450ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31572025-04-30T06:35:37.456ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31582025-04-30T06:35:37.456ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31592025-04-30T06:35:37.456ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31602025-04-30T06:35:37.456ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31612025-04-30T06:35:37.462ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
31622025-04-30T06:35:37.465ZDEBGtest_metrics (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
31632025-04-30T06:35:37.465ZTRACtest_metrics (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [SchemeName("spoof")] }) local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 uri = /v1/system/ip-pools
31642025-04-30T06:35:37.465ZTRACtest_metrics (dropshot_external): authorize begin action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource = IpPoolList uri = /v1/system/ip-pools
31652025-04-30T06:35:37.465ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet uri = /v1/system/ip-pools
31662025-04-30T06:35:37.465ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource = Database uri = /v1/system/ip-pools
31672025-04-30T06:35:37.465ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 roles = RoleSet { roles: {} } uri = /v1/system/ip-pools
31682025-04-30T06:35:37.465ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource = Database result = Ok(()) uri = /v1/system/ip-pools
31692025-04-30T06:35:37.474ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } uri = /v1/system/ip-pools
31702025-04-30T06:35:37.479ZDEBGtest_metrics (dropshot_external): authorize result action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource = IpPoolList result = Ok(()) uri = /v1/system/ip-pools
31712025-04-30T06:35:37.479ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource = Database uri = /v1/system/ip-pools
31722025-04-30T06:35:37.479ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 roles = RoleSet { roles: {} } uri = /v1/system/ip-pools
31732025-04-30T06:35:37.480ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 resource = Database result = Ok(()) uri = /v1/system/ip-pools
31742025-04-30T06:35:37.484ZINFOtest_metrics (dropshot_external): request completed latency_us = 116983 local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 66e2932f-9b03-4d73-a112-a8176e2b70e5 response_code = 201 uri = /v1/system/ip-pools
31752025-04-30T06:35:37.485ZINFOtest_metrics (external client test context): client received response status = 201
31762025-04-30T06:35:37.486ZINFOtest_metrics (external client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:38219/v1/system/ip-pools/default/ranges/add
31772025-04-30T06:35:37.486ZTRACtest_metrics (dropshot_external): incoming request local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe uri = /v1/system/ip-pools/default/ranges/add
31782025-04-30T06:35:37.487ZTRACtest_metrics (dropshot_external): authn: trying SchemeName("spoof") local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe uri = /v1/system/ip-pools/default/ranges/add
31792025-04-30T06:35:37.487ZDEBGtest_metrics (dropshot_external): looking up silo for user local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/system/ip-pools/default/ranges/add
31802025-04-30T06:35:37.487ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31812025-04-30T06:35:37.487ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31822025-04-30T06:35:37.488ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31832025-04-30T06:35:37.491ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31842025-04-30T06:35:37.491ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31852025-04-30T06:35:37.492ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31862025-04-30T06:35:37.495ZTRACtest_metrics (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007, lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
31872025-04-30T06:35:37.495ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
31882025-04-30T06:35:37.495ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31892025-04-30T06:35:37.495ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31902025-04-30T06:35:37.495ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31912025-04-30T06:35:37.498ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
31922025-04-30T06:35:37.498ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31932025-04-30T06:35:37.498ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31942025-04-30T06:35:37.499ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
31952025-04-30T06:35:37.502ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
31962025-04-30T06:35:37.520ZDEBGtest_metrics (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007, lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
31972025-04-30T06:35:37.520ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
31982025-04-30T06:35:37.520ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
31992025-04-30T06:35:37.520ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32002025-04-30T06:35:37.525ZTRACtest_metrics (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
32012025-04-30T06:35:37.525ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
32022025-04-30T06:35:37.525ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32032025-04-30T06:35:37.525ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32042025-04-30T06:35:37.526ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32052025-04-30T06:35:37.529ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32062025-04-30T06:35:37.529ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32072025-04-30T06:35:37.529ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32082025-04-30T06:35:37.530ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32092025-04-30T06:35:37.534ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
32102025-04-30T06:35:37.538ZDEBGtest_metrics (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
32112025-04-30T06:35:37.538ZTRACtest_metrics (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [SchemeName("spoof")] }) local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe uri = /v1/system/ip-pools/default/ranges/add
32122025-04-30T06:35:37.539ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database uri = /v1/system/ip-pools/default/ranges/add
32132025-04-30T06:35:37.539ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/ranges/add
32142025-04-30T06:35:37.539ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32152025-04-30T06:35:37.541ZTRACtest_metrics (dropshot_external): authorize begin action = Modify actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = IpPool { parent: Fleet, key: c1dff3ab-da07-43d8-bf25-f2b1c6eb04ca, lookup_type: ByName("default") } uri = /v1/system/ip-pools/default/ranges/add
32162025-04-30T06:35:37.541ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet uri = /v1/system/ip-pools/default/ranges/add
32172025-04-30T06:35:37.541ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database uri = /v1/system/ip-pools/default/ranges/add
32182025-04-30T06:35:37.541ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/ranges/add
32192025-04-30T06:35:37.549ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32202025-04-30T06:35:37.553ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } uri = /v1/system/ip-pools/default/ranges/add
32212025-04-30T06:35:37.557ZDEBGtest_metrics (dropshot_external): authorize result action = Modify actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = IpPool { parent: Fleet, key: c1dff3ab-da07-43d8-bf25-f2b1c6eb04ca, lookup_type: ByName("default") } result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32222025-04-30T06:35:37.557ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database uri = /v1/system/ip-pools/default/ranges/add
32232025-04-30T06:35:37.557ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/ranges/add
32242025-04-30T06:35:37.558ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32252025-04-30T06:35:37.563ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database uri = /v1/system/ip-pools/default/ranges/add
32262025-04-30T06:35:37.564ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/ranges/add
32272025-04-30T06:35:37.565ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32282025-04-30T06:35:37.571ZTRACtest_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32292025-04-30T06:35:37.571ZTRACtest_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
32302025-04-30T06:35:37.571ZTRACtest_metrics (dropshot_external): authorize begin action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = IpPool { parent: Fleet, key: c1dff3ab-da07-43d8-bf25-f2b1c6eb04ca, lookup_type: ByName("default") } uri = /v1/system/ip-pools/default/ranges/add
32312025-04-30T06:35:37.571ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet uri = /v1/system/ip-pools/default/ranges/add
32322025-04-30T06:35:37.572ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database uri = /v1/system/ip-pools/default/ranges/add
32332025-04-30T06:35:37.572ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/ranges/add
32342025-04-30T06:35:37.573ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32352025-04-30T06:35:37.578ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } uri = /v1/system/ip-pools/default/ranges/add
32362025-04-30T06:35:37.580ZDEBGtest_metrics (dropshot_external): authorize result action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe resource = IpPool { parent: Fleet, key: c1dff3ab-da07-43d8-bf25-f2b1c6eb04ca, lookup_type: ByName("default") } result = Ok(()) uri = /v1/system/ip-pools/default/ranges/add
32372025-04-30T06:35:37.588ZINFOtest_metrics (dropshot_external): request completed latency_us = 101695 local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = 755e1b5c-ffe4-4c47-9180-1fbf780fcdbe response_code = 201 uri = /v1/system/ip-pools/default/ranges/add
32382025-04-30T06:35:37.589ZINFOtest_metrics (external client test context): client received response status = 201
32392025-04-30T06:35:37.589ZINFOtest_metrics (external client test context): client request body = Body { inner: BoxBody } method = POST uri = http://127.0.0.1:38219/v1/system/ip-pools/default/silos
32402025-04-30T06:35:37.590ZTRACtest_metrics (dropshot_external): incoming request local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb uri = /v1/system/ip-pools/default/silos
32412025-04-30T06:35:37.590ZTRACtest_metrics (dropshot_external): authn: trying SchemeName("spoof") local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb uri = /v1/system/ip-pools/default/silos
32422025-04-30T06:35:37.590ZDEBGtest_metrics (dropshot_external): looking up silo for user local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb silo_user_id = 001de000-05e4-4000-8000-000000004007 uri = /v1/system/ip-pools/default/silos
32432025-04-30T06:35:37.590ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32442025-04-30T06:35:37.590ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32452025-04-30T06:35:37.591ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32462025-04-30T06:35:37.593ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32472025-04-30T06:35:37.593ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32482025-04-30T06:35:37.594ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32492025-04-30T06:35:37.598ZTRACtest_metrics (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007, lookup_type: ById(001de000-05e4-4000-8000-000000004007) }
32502025-04-30T06:35:37.598ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
32512025-04-30T06:35:37.598ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32522025-04-30T06:35:37.598ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32532025-04-30T06:35:37.600ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32542025-04-30T06:35:37.603ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32552025-04-30T06:35:37.603ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32562025-04-30T06:35:37.603ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32572025-04-30T06:35:37.603ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32582025-04-30T06:35:37.606ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
32592025-04-30T06:35:37.610ZDEBGtest_metrics (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = SiloUser { parent: Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }, key: 001de000-05e4-4000-8000-000000004007, lookup_type: ById(001de000-05e4-4000-8000-000000004007) } result = Ok(())
32602025-04-30T06:35:37.610ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32612025-04-30T06:35:37.610ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32622025-04-30T06:35:37.611ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32632025-04-30T06:35:37.613ZTRACtest_metrics (ExternalAuthn): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) }
32642025-04-30T06:35:37.613ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo
32652025-04-30T06:35:37.613ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32662025-04-30T06:35:37.613ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32672025-04-30T06:35:37.614ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32682025-04-30T06:35:37.617ZTRACtest_metrics (ExternalAuthn): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
32692025-04-30T06:35:37.617ZTRACtest_metrics (ExternalAuthn): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database
32702025-04-30T06:35:37.617ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {} }
32712025-04-30T06:35:37.618ZDEBGtest_metrics (ExternalAuthn): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Database result = Ok(())
32722025-04-30T06:35:37.620ZDEBGtest_metrics (ExternalAuthn): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "external-authenticator")} }
32732025-04-30T06:35:37.623ZDEBGtest_metrics (ExternalAuthn): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000003, .. }) resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(())
32742025-04-30T06:35:37.623ZTRACtest_metrics (dropshot_external): authn result: Ok(Context { kind: Authenticated(Details { actor: Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } }, Some(SiloAuthnPolicy { mapped_fleet_roles: {} })), schemes_tried: [SchemeName("spoof")] }) local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb uri = /v1/system/ip-pools/default/silos
32752025-04-30T06:35:37.623ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
32762025-04-30T06:35:37.623ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
32772025-04-30T06:35:37.624ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
32782025-04-30T06:35:37.627ZTRACtest_metrics (dropshot_external): authorize begin action = Modify actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = IpPool { parent: Fleet, key: c1dff3ab-da07-43d8-bf25-f2b1c6eb04ca, lookup_type: ByName("default") } uri = /v1/system/ip-pools/default/silos
32792025-04-30T06:35:37.628ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet uri = /v1/system/ip-pools/default/silos
32802025-04-30T06:35:37.628ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
32812025-04-30T06:35:37.628ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
32822025-04-30T06:35:37.628ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
32832025-04-30T06:35:37.636ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } uri = /v1/system/ip-pools/default/silos
32842025-04-30T06:35:37.641ZDEBGtest_metrics (dropshot_external): authorize result action = Modify actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = IpPool { parent: Fleet, key: c1dff3ab-da07-43d8-bf25-f2b1c6eb04ca, lookup_type: ByName("default") } result = Ok(()) uri = /v1/system/ip-pools/default/silos
32852025-04-30T06:35:37.641ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
32862025-04-30T06:35:37.641ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
32872025-04-30T06:35:37.642ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
32882025-04-30T06:35:37.647ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
32892025-04-30T06:35:37.647ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
32902025-04-30T06:35:37.648ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
32912025-04-30T06:35:37.650ZTRACtest_metrics (dropshot_external): authorize begin action = Modify actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } uri = /v1/system/ip-pools/default/silos
32922025-04-30T06:35:37.651ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource_id = 001de000-5110-4000-8000-000000000000 resource_type = Silo uri = /v1/system/ip-pools/default/silos
32932025-04-30T06:35:37.651ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
32942025-04-30T06:35:37.651ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
32952025-04-30T06:35:37.651ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
32962025-04-30T06:35:37.654ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet uri = /v1/system/ip-pools/default/silos
32972025-04-30T06:35:37.654ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
32982025-04-30T06:35:37.654ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
32992025-04-30T06:35:37.655ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
33002025-04-30T06:35:37.659ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin"), (Silo, 001de000-5110-4000-8000-000000000000, "admin")} } uri = /v1/system/ip-pools/default/silos
33012025-04-30T06:35:37.662ZDEBGtest_metrics (dropshot_external): authorize result action = Modify actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Silo { parent: Fleet, key: 001de000-5110-4000-8000-000000000000, lookup_type: ById(001de000-5110-4000-8000-000000000000) } result = Ok(()) uri = /v1/system/ip-pools/default/silos
33022025-04-30T06:35:37.662ZTRACtest_metrics (dropshot_external): authorize begin action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = IpPoolList uri = /v1/system/ip-pools/default/silos
33032025-04-30T06:35:37.662ZTRACtest_metrics (dropshot_external): loading roles actor = Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. } actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet uri = /v1/system/ip-pools/default/silos
33042025-04-30T06:35:37.662ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
33052025-04-30T06:35:37.662ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
33062025-04-30T06:35:37.663ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
33072025-04-30T06:35:37.667ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} } uri = /v1/system/ip-pools/default/silos
33082025-04-30T06:35:37.671ZDEBGtest_metrics (dropshot_external): authorize result action = CreateChild actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = IpPoolList result = Ok(()) uri = /v1/system/ip-pools/default/silos
33092025-04-30T06:35:37.671ZTRACtest_metrics (dropshot_external): authorize begin action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database uri = /v1/system/ip-pools/default/silos
33102025-04-30T06:35:37.671ZDEBGtest_metrics (dropshot_external): roles actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb roles = RoleSet { roles: {} } uri = /v1/system/ip-pools/default/silos
33112025-04-30T06:35:37.671ZDEBGtest_metrics (dropshot_external): authorize result action = Query actor = Some(Actor::SiloUser { silo_user_id: 001de000-05e4-4000-8000-000000004007, silo_id: 001de000-5110-4000-8000-000000000000, .. }) actor_id = 001de000-05e4-4000-8000-000000004007 authenticated = true local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb resource = Database result = Ok(()) uri = /v1/system/ip-pools/default/silos
33122025-04-30T06:35:37.713ZINFOtest_metrics (dropshot_external): request completed latency_us = 123679 local_addr = 127.0.0.1:38219 method = POST remote_addr = 127.0.0.1:64657 req_id = ce9328d8-9408-47c0-8e5a-f41a1de0cfdb response_code = 201 uri = /v1/system/ip-pools/default/silos
33132025-04-30T06:35:37.714ZINFOtest_metrics (external client test context): client received response status = 201
33142025-04-30T06:35:37.715ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = GET path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33152025-04-30T06:35:37.715ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = POST path = /crucible/0/regions server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33162025-04-30T06:35:37.715ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = DELETE path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33172025-04-30T06:35:37.715ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = GET path = /crucible/0/regions/{id} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33182025-04-30T06:35:37.715ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = GET path = /crucible/0/regions/{id}/snapshots server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33192025-04-30T06:35:37.715ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33202025-04-30T06:35:37.716ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = GET path = /crucible/0/regions/{id}/snapshots/{name} server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33212025-04-30T06:35:37.716ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = DELETE path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33222025-04-30T06:35:37.716ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): registered endpoint kind = storage local_addr = [::1]:36196 method = POST path = /crucible/0/regions/{id}/snapshots/{name}/run server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 versions = all
33232025-04-30T06:35:37.716ZINFOtest_metrics (Simulated CrucibleAgent Dropshot Server): listening kind = storage local_addr = [::1]:36196 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33242025-04-30T06:35:37.716ZDEBGtest_metrics (Simulated CrucibleAgent Dropshot Server): successfully registered DTrace USDT probes kind = storage local_addr = [::1]:36196 server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33252025-04-30T06:35:37.717ZINFOtest_metrics (SledAgent): Created Simulated Crucible Server address = [::1]:36196 kind = storage server = b6d65341-167c-41df-9b5c-41cded99c229 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
33262025-04-30T06:35:37.717ZINFOtest_metrics (nexus): inserting test physical disk model = test-model physical_disk_id = 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 serial = totally-unique-serial: 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
33272025-04-30T06:35:37.717ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33282025-04-30T06:35:37.717ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33292025-04-30T06:35:37.717ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33302025-04-30T06:35:37.720ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33312025-04-30T06:35:37.721ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33322025-04-30T06:35:37.721ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33332025-04-30T06:35:37.722ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
33342025-04-30T06:35:37.722ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33352025-04-30T06:35:37.723ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33362025-04-30T06:35:37.723ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33372025-04-30T06:35:37.723ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33382025-04-30T06:35:37.726ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33392025-04-30T06:35:37.730ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
33402025-04-30T06:35:37.835ZDEBGtest_metrics (producer-registration-task): will register with Nexus at 127.0.0.1:49095
33412025-04-30T06:35:37.950ZDEBGtest_metrics (producer-registration-task): client request body = Some(Body) method = POST uri = http://127.0.0.1:49095/metrics/producers
33422025-04-30T06:35:37.952ZINFOtest_metrics (nexus): upserting zpool physical_disk_id = 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 9eb65229-6110-4f47-932d-e63d3ee0027d
33432025-04-30T06:35:37.952ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33442025-04-30T06:35:37.952ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33452025-04-30T06:35:37.952ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33462025-04-30T06:35:37.952ZINFOtest_metrics (dropshot_internal): accepted connection local_addr = 127.0.0.1:49095 remote_addr = 127.0.0.1:48521
33472025-04-30T06:35:37.953ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:48521 req_id = 6e04b23b-9db4-453c-adba-b09890e9ce33 uri = /metrics/producers
33482025-04-30T06:35:37.953ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:48521 req_id = 6e04b23b-9db4-453c-adba-b09890e9ce33 resource = Database uri = /metrics/producers
33492025-04-30T06:35:37.954ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:48521 req_id = 6e04b23b-9db4-453c-adba-b09890e9ce33 roles = RoleSet { roles: {} } uri = /metrics/producers
33502025-04-30T06:35:37.954ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:48521 req_id = 6e04b23b-9db4-453c-adba-b09890e9ce33 resource = Database result = Ok(()) uri = /metrics/producers
33512025-04-30T06:35:37.954ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
33522025-04-30T06:35:37.955ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33532025-04-30T06:35:37.955ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33542025-04-30T06:35:37.955ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33552025-04-30T06:35:37.956ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33562025-04-30T06:35:37.971ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33572025-04-30T06:35:37.983ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
33582025-04-30T06:35:37.984ZINFOtest_metrics (nexus): assigned collector to new producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
33592025-04-30T06:35:37.984ZINFOtest_metrics (dropshot_internal): request completed latency_us = 30964 local_addr = 127.0.0.1:49095 method = POST remote_addr = 127.0.0.1:48521 req_id = 6e04b23b-9db4-453c-adba-b09890e9ce33 response_code = 201 uri = /metrics/producers
33602025-04-30T06:35:37.985ZDEBGtest_metrics (producer-registration-task): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/producers", status: 201, headers: {"content-type": "application/json", "x-request-id": "6e04b23b-9db4-453c-adba-b09890e9ce33", "content-length": "41", "date": "Wed, 30 Apr 2025 06:35:37 GMT"} })
33612025-04-30T06:35:37.995ZDEBGtest_metrics (producer-registration-task): successfully registered with Nexus, pausing until time to renew lease lease_duration = 600s wait_period = 150s
33622025-04-30T06:35:37.996ZINFOtest_metrics (nexus): upserting Crucible dataset dataset_id = 75aa51a9-b732-4a5d-9e09-59fad2a4921f zpool_id = 9eb65229-6110-4f47-932d-e63d3ee0027d
33632025-04-30T06:35:38.009ZINFOtest_metrics (nexus): inserting test physical disk model = test-model physical_disk_id = 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 serial = totally-unique-serial: 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 vendor = test-vendor
33642025-04-30T06:35:38.010ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33652025-04-30T06:35:38.010ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33662025-04-30T06:35:38.010ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33672025-04-30T06:35:38.015ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 (physical_disk), lookup_type: ById(7b8de7c0-f4d1-4b5e-9e30-22021253fb31) }
33682025-04-30T06:35:38.018ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33692025-04-30T06:35:38.018ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33702025-04-30T06:35:38.018ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33712025-04-30T06:35:38.019ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33722025-04-30T06:35:38.028ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33732025-04-30T06:35:38.035ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = PhysicalDisk { parent: Fleet, key: 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 (physical_disk), lookup_type: ById(7b8de7c0-f4d1-4b5e-9e30-22021253fb31) } result = Ok(())
33742025-04-30T06:35:38.035ZINFOtest_metrics (nexus): upserting zpool physical_disk_id = 7b8de7c0-f4d1-4b5e-9e30-22021253fb31 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 zpool_id = 9eb65229-6110-4f47-932d-e63d3ee0027d
33752025-04-30T06:35:38.035ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33762025-04-30T06:35:38.035ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33772025-04-30T06:35:38.035ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33782025-04-30T06:35:38.036ZTRACtest_metrics (InternalApi): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
33792025-04-30T06:35:38.037ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33802025-04-30T06:35:38.037ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33812025-04-30T06:35:38.037ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33822025-04-30T06:35:38.038ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33832025-04-30T06:35:38.075ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33842025-04-30T06:35:38.078ZDEBGtest_metrics (InternalApi): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
33852025-04-30T06:35:38.085ZTRACtest_metrics (InternalApi): authorize begin action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet
33862025-04-30T06:35:38.085ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33872025-04-30T06:35:38.085ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33882025-04-30T06:35:38.085ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33892025-04-30T06:35:38.086ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33902025-04-30T06:35:38.091ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
33912025-04-30T06:35:38.094ZDEBGtest_metrics (InternalApi): authorize result action = CreateChild actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Fleet result = Ok(())
33922025-04-30T06:35:38.095ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33932025-04-30T06:35:38.095ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33942025-04-30T06:35:38.095ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
33952025-04-30T06:35:38.109ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
33962025-04-30T06:35:38.109ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
33972025-04-30T06:35:38.109ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
33982025-04-30T06:35:38.109ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
33992025-04-30T06:35:38.111ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34002025-04-30T06:35:38.111ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 4 reason = Signaled
34012025-04-30T06:35:38.112ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
34022025-04-30T06:35:38.112ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34032025-04-30T06:35:38.112ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34042025-04-30T06:35:38.112ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34052025-04-30T06:35:38.112ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34062025-04-30T06:35:38.116ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34072025-04-30T06:35:38.132ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34082025-04-30T06:35:38.132ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34092025-04-30T06:35:38.132ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34102025-04-30T06:35:38.133ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34112025-04-30T06:35:38.136ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34122025-04-30T06:35:38.141ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
34132025-04-30T06:35:38.141ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
34142025-04-30T06:35:38.141ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
34152025-04-30T06:35:38.142ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
34162025-04-30T06:35:38.143ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34172025-04-30T06:35:38.144ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34182025-04-30T06:35:38.146ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34192025-04-30T06:35:38.146ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
34202025-04-30T06:35:38.146ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
34212025-04-30T06:35:38.150ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ad 2f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ./..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:54589 req_id = ffd8c054-1e93-47cf-aeef-b8e8f5b7434a
34222025-04-30T06:35:38.150ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44335,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54589 req_id = ffd8c054-1e93-47cf-aeef-b8e8f5b7434a
34232025-04-30T06:35:38.151ZDEBGtest_metrics (store): query key key = _mgs._tcp
34242025-04-30T06:35:38.158ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34252025-04-30T06:35:38.162ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54589 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ffd8c054-1e93-47cf-aeef-b8e8f5b7434a
34262025-04-30T06:35:38.163ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3802, tv_nsec: 962211563 } })
34272025-04-30T06:35:38.163ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
34282025-04-30T06:35:38.165ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 00 f3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59141 req_id = 2e7d2451-3f81-407a-99d6-759d891456de
34292025-04-30T06:35:38.165ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 243,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59141 req_id = 2e7d2451-3f81-407a-99d6-759d891456de
34302025-04-30T06:35:38.165ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
34312025-04-30T06:35:38.165ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59141 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2e7d2451-3f81-407a-99d6-759d891456de
34322025-04-30T06:35:38.290ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
34332025-04-30T06:35:38.291ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d3 f6 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56616 req_id = d7b29d7c-5ee0-4168-a79a-43d900ecb1dc
34342025-04-30T06:35:38.291ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54262,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56616 req_id = d7b29d7c-5ee0-4168-a79a-43d900ecb1dc
34352025-04-30T06:35:38.291ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
34362025-04-30T06:35:38.291ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 54262, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56616 req_id = d7b29d7c-5ee0-4168-a79a-43d900ecb1dc
34372025-04-30T06:35:38.291ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
34382025-04-30T06:35:38.291ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
34392025-04-30T06:35:38.297ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
34402025-04-30T06:35:38.318ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9883d920-fd1a-4267-a9d5-55d1f8e1250f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34412025-04-30T06:35:38.319ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
34422025-04-30T06:35:38.322ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8f14de0-b5ef-4350-8e0c-a279bccdf016", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34432025-04-30T06:35:38.322ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
34442025-04-30T06:35:38.327ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e2f03f7-6ab0-4b24-b351-45d5a512be4e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34452025-04-30T06:35:38.327ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
34462025-04-30T06:35:38.366ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31a30bc6-b574-4c63-94f0-4ab5729ac755", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34472025-04-30T06:35:38.366ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
34482025-04-30T06:35:38.379ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
34492025-04-30T06:35:38.406ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f76972c6-4308-4745-bf27-ffeb72f5ca1a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34502025-04-30T06:35:38.406ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
34512025-04-30T06:35:38.439ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
34522025-04-30T06:35:38.440ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
34532025-04-30T06:35:38.440ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34542025-04-30T06:35:38.440ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34552025-04-30T06:35:38.442ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34562025-04-30T06:35:38.442ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97da907b-85d9-4f5c-af33-ef7b404764b9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34572025-04-30T06:35:38.442ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
34582025-04-30T06:35:38.445ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
34592025-04-30T06:35:38.451ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
34602025-04-30T06:35:38.451ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34612025-04-30T06:35:38.451ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34622025-04-30T06:35:38.452ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34632025-04-30T06:35:38.454ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "885a787d-384c-4947-a73a-1596c76d9402", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34642025-04-30T06:35:38.454ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
34652025-04-30T06:35:38.455ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
34662025-04-30T06:35:38.455ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
34672025-04-30T06:35:38.456ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
34682025-04-30T06:35:38.459ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1017d62b-c378-4dec-b920-c894a5ff5878", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34692025-04-30T06:35:38.459ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
34702025-04-30T06:35:38.461ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af642f08-ab05-47dd-91c9-9a9bcb280bc3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34712025-04-30T06:35:38.461ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
34722025-04-30T06:35:38.462ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9305390d-8af0-442e-880c-d5d82be351e2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34732025-04-30T06:35:38.462ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
34742025-04-30T06:35:38.464ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe54869b-c5b4-4fe1-9e01-1ee70986d84a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34752025-04-30T06:35:38.464ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
34762025-04-30T06:35:38.465ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3ae0ff1-55f4-4661-a857-71d680800661", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34772025-04-30T06:35:38.465ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
34782025-04-30T06:35:38.466ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b319f15-296d-4ac3-8aae-011534b7d8d4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34792025-04-30T06:35:38.467ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
34802025-04-30T06:35:38.470ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4a3a44c-67d3-436e-9b71-bb0350baf284", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34812025-04-30T06:35:38.470ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
34822025-04-30T06:35:38.472ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c1020c7-557f-4ff4-9d07-d7efe850fafc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34832025-04-30T06:35:38.472ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
34842025-04-30T06:35:38.476ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6b36aa5-e2fa-4c62-a99a-49dad9b71ccd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34852025-04-30T06:35:38.477ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
34862025-04-30T06:35:38.480ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3faa157-72b5-47b3-8601-316e8ed36fdf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34872025-04-30T06:35:38.480ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
34882025-04-30T06:35:38.484ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9509189e-1f2e-4165-89e7-60882da2fa87", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34892025-04-30T06:35:38.484ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
34902025-04-30T06:35:38.491ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1199d1ea-d420-4358-8ea7-38c92c096a35", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34912025-04-30T06:35:38.491ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
34922025-04-30T06:35:38.492ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8a8830c-7055-4109-8c53-ad964538fa53", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34932025-04-30T06:35:38.493ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
34942025-04-30T06:35:38.494ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11785e2e-c7bc-49aa-9d91-ca03a6cc8014", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34952025-04-30T06:35:38.494ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
34962025-04-30T06:35:38.496ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f561e67-849d-46c2-bc92-b5ad00dd584e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34972025-04-30T06:35:38.496ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
34982025-04-30T06:35:38.497ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75537746-49cf-41b5-9544-01001820d43c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
34992025-04-30T06:35:38.498ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
35002025-04-30T06:35:38.503ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f3c668-b5b2-4242-bf7e-bc30b3eea32f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35012025-04-30T06:35:38.504ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
35022025-04-30T06:35:38.505ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
35032025-04-30T06:35:38.508ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "121b668d-3604-47be-9364-e608af47c53b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35042025-04-30T06:35:38.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
35052025-04-30T06:35:38.510ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d11428c3-2955-4cb2-bfcb-0f42fbe4a72a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35062025-04-30T06:35:38.510ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
35072025-04-30T06:35:38.513ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30f2aa36-f464-4611-bfef-80b8a8b70366", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35082025-04-30T06:35:38.513ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
35092025-04-30T06:35:38.516ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40aa409-9113-4e43-865a-de3c8777b931", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35102025-04-30T06:35:38.516ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
35112025-04-30T06:35:38.519ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49525884-9f9e-490f-85fa-5757ac90d871", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35122025-04-30T06:35:38.525ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
35132025-04-30T06:35:38.528ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39436332-d7c0-4cab-a5e5-b57bbef1a986", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35142025-04-30T06:35:38.528ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
35152025-04-30T06:35:38.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc557afe-0dd4-473b-a371-dcba5969ef81", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35162025-04-30T06:35:38.530ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
35172025-04-30T06:35:38.531ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f30f349-8af1-4584-a36c-1f2700a052f1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35182025-04-30T06:35:38.531ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
35192025-04-30T06:35:38.532ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "544c0fa5-4a96-4f22-ba69-1d780efc267d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35202025-04-30T06:35:38.533ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
35212025-04-30T06:35:38.534ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "957a21d9-1f6d-4fff-a4cf-598d27f0dd60", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35222025-04-30T06:35:38.534ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
35232025-04-30T06:35:38.536ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b2d4634-8e83-4b7c-ac45-edb3571e3f43", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35242025-04-30T06:35:38.536ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
35252025-04-30T06:35:38.538ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "081314c2-dbae-4b9f-8f3b-45852f6e6f59", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35262025-04-30T06:35:38.538ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
35272025-04-30T06:35:38.540ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "710cc28c-f363-4029-baae-ceb7856508e0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35282025-04-30T06:35:38.541ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
35292025-04-30T06:35:38.543ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb161733-bfbe-4647-baa7-3eb73e16b572", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35302025-04-30T06:35:38.544ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
35312025-04-30T06:35:38.546ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "250b0ef7-b1ba-4b5b-8527-67853e235fe3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35322025-04-30T06:35:38.546ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
35332025-04-30T06:35:38.548ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bc410ec-3c85-4dfb-b6ae-5cf0ac7d4672", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35342025-04-30T06:35:38.548ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
35352025-04-30T06:35:38.550ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d13c50-0b2f-45e6-971f-8d8fc6cba7dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35362025-04-30T06:35:38.550ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
35372025-04-30T06:35:38.551ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dbd1d305-5cf4-4e6b-bf8a-75d61731096f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35382025-04-30T06:35:38.551ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
35392025-04-30T06:35:38.553ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ed19c4-2562-41a4-b5ee-c0b475628156", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35402025-04-30T06:35:38.571ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
35412025-04-30T06:35:38.571ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35422025-04-30T06:35:38.571ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35432025-04-30T06:35:38.571ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35442025-04-30T06:35:38.571ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35452025-04-30T06:35:38.572ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35462025-04-30T06:35:38.572ZTRACtest_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35472025-04-30T06:35:38.573ZTRACtest_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
35482025-04-30T06:35:38.574ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df2448cc-7790-490a-8125-6dba90d32d85", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35492025-04-30T06:35:38.574ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
35502025-04-30T06:35:38.576ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35512025-04-30T06:35:38.582ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
35522025-04-30T06:35:38.582ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35532025-04-30T06:35:38.582ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35542025-04-30T06:35:38.583ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35552025-04-30T06:35:38.587ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e2ce9f1-eb98-4021-bd2e-ee492f403c1c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35562025-04-30T06:35:38.589ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35572025-04-30T06:35:38.589ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35582025-04-30T06:35:38.589ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35592025-04-30T06:35:38.589ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35602025-04-30T06:35:38.589ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35612025-04-30T06:35:38.590ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35622025-04-30T06:35:38.590ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
35632025-04-30T06:35:38.590ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
35642025-04-30T06:35:38.593ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35652025-04-30T06:35:38.600ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35662025-04-30T06:35:38.600ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
35672025-04-30T06:35:38.600ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35682025-04-30T06:35:38.600ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35692025-04-30T06:35:38.600ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35702025-04-30T06:35:38.601ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35712025-04-30T06:35:38.609ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35722025-04-30T06:35:38.617ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
35732025-04-30T06:35:38.617ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35742025-04-30T06:35:38.617ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35752025-04-30T06:35:38.618ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35762025-04-30T06:35:38.696ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
35772025-04-30T06:35:38.696ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
35782025-04-30T06:35:38.696ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:38699 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
35792025-04-30T06:35:38.697ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:38699 req_id = 8a8c15cc-6923-4e55-a6cd-9858b8c65072 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35802025-04-30T06:35:38.697ZINFOtest_metrics (dropshot): request completed latency_us = 542 local_addr = [::1]:56237 method = GET remote_addr = [::1]:38699 req_id = 8a8c15cc-6923-4e55-a6cd-9858b8c65072 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
35812025-04-30T06:35:38.698ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a8c15cc-6923-4e55-a6cd-9858b8c65072", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
35822025-04-30T06:35:38.699ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35832025-04-30T06:35:38.699ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
35842025-04-30T06:35:38.699ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
35852025-04-30T06:35:38.704ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
35862025-04-30T06:35:38.705ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35872025-04-30T06:35:38.705ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35882025-04-30T06:35:38.705ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35892025-04-30T06:35:38.706ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35902025-04-30T06:35:38.711ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
35912025-04-30T06:35:38.717ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
35922025-04-30T06:35:38.717ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
35932025-04-30T06:35:38.717ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
35942025-04-30T06:35:38.718ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
35952025-04-30T06:35:38.763ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
35962025-04-30T06:35:38.814ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
35972025-04-30T06:35:38.815ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
35982025-04-30T06:35:38.815ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
35992025-04-30T06:35:38.815ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36002025-04-30T06:35:38.816ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36012025-04-30T06:35:38.819ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36022025-04-30T06:35:38.827ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36032025-04-30T06:35:38.827ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36042025-04-30T06:35:38.827ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36052025-04-30T06:35:38.827ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36062025-04-30T06:35:38.868ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36072025-04-30T06:35:38.868ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36082025-04-30T06:35:38.869ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36092025-04-30T06:35:38.870ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
36102025-04-30T06:35:38.871ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
36112025-04-30T06:35:38.871ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 759.168724ms iteration = 4 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
36122025-04-30T06:35:38.871ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 5 reason = Signaled
36132025-04-30T06:35:38.871ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
36142025-04-30T06:35:38.871ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36152025-04-30T06:35:38.871ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36162025-04-30T06:35:38.871ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36172025-04-30T06:35:38.871ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36182025-04-30T06:35:38.877ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36192025-04-30T06:35:38.882ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
36202025-04-30T06:35:38.882ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
36212025-04-30T06:35:38.882ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
36222025-04-30T06:35:38.883ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
36232025-04-30T06:35:38.891ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
36242025-04-30T06:35:38.891ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
36252025-04-30T06:35:38.892ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 57 62 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Wb..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50832 req_id = 89104a99-3a30-4cdf-8f28-bd660ad15c10
36262025-04-30T06:35:38.897ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22370,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50832 req_id = 89104a99-3a30-4cdf-8f28-bd660ad15c10
36272025-04-30T06:35:38.897ZDEBGtest_metrics (store): query key key = _mgs._tcp
36282025-04-30T06:35:38.897ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36292025-04-30T06:35:38.897ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50832 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 89104a99-3a30-4cdf-8f28-bd660ad15c10
36302025-04-30T06:35:38.898ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3803, tv_nsec: 697071353 } })
36312025-04-30T06:35:38.898ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
36322025-04-30T06:35:38.898ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 41 fe 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e A...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62401 req_id = 992f886c-ba67-4166-aee7-215f86071bd6
36332025-04-30T06:35:38.898ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16894,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62401 req_id = 992f886c-ba67-4166-aee7-215f86071bd6
36342025-04-30T06:35:38.898ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
36352025-04-30T06:35:38.899ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62401 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 992f886c-ba67-4166-aee7-215f86071bd6
36362025-04-30T06:35:38.979ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
36372025-04-30T06:35:38.983ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 55 b5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c U............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61395 req_id = 6c1aa78b-29b2-4e19-a262-972dd2fa753b
36382025-04-30T06:35:38.983ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21941,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61395 req_id = 6c1aa78b-29b2-4e19-a262-972dd2fa753b
36392025-04-30T06:35:38.983ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
36402025-04-30T06:35:38.983ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21941, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61395 req_id = 6c1aa78b-29b2-4e19-a262-972dd2fa753b
36412025-04-30T06:35:38.984ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
36422025-04-30T06:35:38.984ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
36432025-04-30T06:35:38.984ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
36442025-04-30T06:35:38.991ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0ee7ce2-db33-456c-90b0-9174aab2b581", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36452025-04-30T06:35:38.992ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
36462025-04-30T06:35:38.995ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "248fcc09-bc92-4515-bb8b-07c21812be41", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36472025-04-30T06:35:38.995ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
36482025-04-30T06:35:39.003ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1c7439d-ca1c-4df6-9b4a-68ccc1e668f3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36492025-04-30T06:35:39.003ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
36502025-04-30T06:35:39.008ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd46de26-2d94-41e3-abf1-a68583150434", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36512025-04-30T06:35:39.008ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
36522025-04-30T06:35:39.012ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "437dbbad-2d4a-4662-82fa-a6b2f90e141a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36532025-04-30T06:35:39.012ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
36542025-04-30T06:35:39.015ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2304d304-46f8-4ae7-b2db-8be6584950bb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36552025-04-30T06:35:39.015ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
36562025-04-30T06:35:39.019ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b559ceac-d851-4d09-bf43-acb3a9676b1d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36572025-04-30T06:35:39.019ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
36582025-04-30T06:35:39.022ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
36592025-04-30T06:35:39.088ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca81c196-b7b0-4bd7-aba0-9da8542e04ae", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36602025-04-30T06:35:39.091ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
36612025-04-30T06:35:39.091ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
36622025-04-30T06:35:39.092ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
36632025-04-30T06:35:39.092ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36642025-04-30T06:35:39.092ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36652025-04-30T06:35:39.097ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36662025-04-30T06:35:39.101ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
36672025-04-30T06:35:39.106ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
36682025-04-30T06:35:39.107ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36692025-04-30T06:35:39.107ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36702025-04-30T06:35:39.107ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36712025-04-30T06:35:39.108ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d80dd6e5-f2a9-4729-b250-19663a02de3f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36722025-04-30T06:35:39.108ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
36732025-04-30T06:35:39.111ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
36742025-04-30T06:35:39.111ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
36752025-04-30T06:35:39.112ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
36762025-04-30T06:35:39.112ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e7c5e0-1953-4a8b-96ea-79e915914da2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36772025-04-30T06:35:39.112ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
36782025-04-30T06:35:39.113ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e40acac-32c8-4b5c-8d88-dada0cc488cb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36792025-04-30T06:35:39.114ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
36802025-04-30T06:35:39.115ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b790b408-108c-412b-83d9-4f732b5d0d42", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36812025-04-30T06:35:39.115ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
36822025-04-30T06:35:39.117ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a28558b-b3fb-436d-a960-b7a0f81cdfeb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36832025-04-30T06:35:39.118ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
36842025-04-30T06:35:39.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99ccf554-ef1b-469e-8eae-0af53fc230a9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36852025-04-30T06:35:39.122ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
36862025-04-30T06:35:39.124ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9c672cc-47b1-4508-babc-aef07b27e6ec", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36872025-04-30T06:35:39.124ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
36882025-04-30T06:35:39.126ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d20796c-0612-4c9b-8384-4772851a30a1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36892025-04-30T06:35:39.127ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
36902025-04-30T06:35:39.131ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "819451ab-a888-4ed6-bcfc-51b93d5b8e5f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36912025-04-30T06:35:39.131ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
36922025-04-30T06:35:39.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46ec4a8d-1204-445a-9411-ea5b5367af92", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36932025-04-30T06:35:39.133ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
36942025-04-30T06:35:39.136ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccae5577-983f-46a5-a199-942d06d8717b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36952025-04-30T06:35:39.136ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
36962025-04-30T06:35:39.137ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "90dbd0aa-848a-411f-b357-6967f94c83df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36972025-04-30T06:35:39.137ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
36982025-04-30T06:35:39.138ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea0fc134-23e5-45ff-8282-06b4f6f936f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
36992025-04-30T06:35:39.138ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
37002025-04-30T06:35:39.139ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7f8b7ad-fc80-487a-9fc8-8b0038b08e1c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37012025-04-30T06:35:39.139ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
37022025-04-30T06:35:39.141ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0da6f69-ee3b-4ab6-a4d4-16243ccc5319", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37032025-04-30T06:35:39.141ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
37042025-04-30T06:35:39.142ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b321d9a5-257c-4af7-91b7-109d608532ed", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37052025-04-30T06:35:39.142ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
37062025-04-30T06:35:39.144ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb49233e-2560-4a1d-acc0-26f97ec2c9f0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37072025-04-30T06:35:39.144ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
37082025-04-30T06:35:39.165ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
37092025-04-30T06:35:39.169ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fed9f8e5-fc49-402a-b0b4-304f2d4a826b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37102025-04-30T06:35:39.169ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
37112025-04-30T06:35:39.185ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89b7316c-36a7-49a5-9131-958d2f617326", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37122025-04-30T06:35:39.186ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
37132025-04-30T06:35:39.188ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75e544fb-ad4f-4535-9013-361c24adf02f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37142025-04-30T06:35:39.189ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
37152025-04-30T06:35:39.192ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41df72d2-2f1e-4f40-b363-4e9a51ea7083", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37162025-04-30T06:35:39.192ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
37172025-04-30T06:35:39.195ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "68b47191-2f4d-4531-8cac-62301c990e76", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37182025-04-30T06:35:39.195ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
37192025-04-30T06:35:39.197ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "88877fd3-5b1c-491e-9cf0-11492b5967b9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37202025-04-30T06:35:39.199ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
37212025-04-30T06:35:39.200ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ffec574-5896-48ec-96a9-956437a9aeac", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37222025-04-30T06:35:39.200ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
37232025-04-30T06:35:39.201ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11a2d38e-7a34-419f-8e76-37a888e59ce2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37242025-04-30T06:35:39.201ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
37252025-04-30T06:35:39.203ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4395953-f87d-4717-bbf1-e0892fd87315", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37262025-04-30T06:35:39.203ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
37272025-04-30T06:35:39.205ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b423d45e-6dbe-4acc-9b0c-e270cf53eb32", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37282025-04-30T06:35:39.205ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
37292025-04-30T06:35:39.210ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "62a7081e-ccc2-4b98-8f43-96dc7c738f33", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37302025-04-30T06:35:39.210ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
37312025-04-30T06:35:39.234ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37322025-04-30T06:35:39.234ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37332025-04-30T06:35:39.234ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37342025-04-30T06:35:39.234ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37352025-04-30T06:35:39.235ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37362025-04-30T06:35:39.238ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d62ba23-3613-4ba6-a8af-637fa2d6edaf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37372025-04-30T06:35:39.238ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
37382025-04-30T06:35:39.238ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37392025-04-30T06:35:39.256ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37402025-04-30T06:35:39.257ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37412025-04-30T06:35:39.257ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37422025-04-30T06:35:39.258ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37432025-04-30T06:35:39.262ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37442025-04-30T06:35:39.263ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37452025-04-30T06:35:39.267ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37462025-04-30T06:35:39.280ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51fe75ce-addf-4255-b67a-b54437882dd0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37472025-04-30T06:35:39.280ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
37482025-04-30T06:35:39.283ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fd6f95e-f6d6-4f4e-80ca-14b79a928acb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37492025-04-30T06:35:39.284ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
37502025-04-30T06:35:39.289ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f32ba50-bd0e-4611-a258-ada36fed9a8a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:38 GMT"} })
37512025-04-30T06:35:39.303ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
37522025-04-30T06:35:39.309ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
37532025-04-30T06:35:39.329ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37542025-04-30T06:35:39.344ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 0b7888af-d744-4de7-bcb6-4d36aa02d04b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37552025-04-30T06:35:39.345ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 0b7888af-d744-4de7-bcb6-4d36aa02d04b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37562025-04-30T06:35:39.345ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 0b7888af-d744-4de7-bcb6-4d36aa02d04b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37572025-04-30T06:35:39.346ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 0b7888af-d744-4de7-bcb6-4d36aa02d04b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37582025-04-30T06:35:39.362ZINFOtest_metrics (dropshot_internal): request completed latency_us = 17680 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 0b7888af-d744-4de7-bcb6-4d36aa02d04b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
37592025-04-30T06:35:39.379ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b7888af-d744-4de7-bcb6-4d36aa02d04b", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37602025-04-30T06:35:39.381ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37612025-04-30T06:35:39.392ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a2a52af5-3ec5-4b18-beed-a83f3cd0b941 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37622025-04-30T06:35:39.395ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a2a52af5-3ec5-4b18-beed-a83f3cd0b941 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37632025-04-30T06:35:39.396ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a2a52af5-3ec5-4b18-beed-a83f3cd0b941 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37642025-04-30T06:35:39.397ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a2a52af5-3ec5-4b18-beed-a83f3cd0b941 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37652025-04-30T06:35:39.399ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
37662025-04-30T06:35:39.404ZINFOtest_metrics (dropshot_internal): request completed latency_us = 12463 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a2a52af5-3ec5-4b18-beed-a83f3cd0b941 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
37672025-04-30T06:35:39.407ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2a52af5-3ec5-4b18-beed-a83f3cd0b941", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37682025-04-30T06:35:39.407ZDEBGtest_metrics (oximeter-agent): registered new metric producer address = 127.0.0.1:65131 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
37692025-04-30T06:35:39.507ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
37702025-04-30T06:35:39.507ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
37712025-04-30T06:35:39.507ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37722025-04-30T06:35:39.507ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37732025-04-30T06:35:39.507ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37742025-04-30T06:35:39.508ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37752025-04-30T06:35:39.511ZDEBGtest_metrics (oximeter-agent): registered new metric producer address = [::1]:62217 collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37762025-04-30T06:35:39.580ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
37772025-04-30T06:35:39.581ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
37782025-04-30T06:35:39.591ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
37792025-04-30T06:35:39.591ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37802025-04-30T06:35:39.591ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37812025-04-30T06:35:39.592ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37822025-04-30T06:35:39.592ZTRACtest_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
37832025-04-30T06:35:39.592ZTRACtest_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
37842025-04-30T06:35:39.595ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
37852025-04-30T06:35:39.602ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
37862025-04-30T06:35:39.602ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
37872025-04-30T06:35:39.604ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
37882025-04-30T06:35:39.617ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e42f0dd5-88f1-4e7a-9ccf-611eac4eb206", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37892025-04-30T06:35:39.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
37902025-04-30T06:35:39.632ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4a14b7c-a1a1-4c8f-82f1-231efba82dbc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37912025-04-30T06:35:39.633ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
37922025-04-30T06:35:39.639ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b88e4fe-d619-4eae-8b1e-d6bb168b4f63", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37932025-04-30T06:35:39.640ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
37942025-04-30T06:35:39.641ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0694c8a4-a346-4ce6-ab95-be1fb9f8339e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37952025-04-30T06:35:39.643ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
37962025-04-30T06:35:39.664ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dc14d0c-e1e3-48d9-9ccf-18c0e3d91749", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
37972025-04-30T06:35:39.664ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
37982025-04-30T06:35:39.665ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
37992025-04-30T06:35:39.665ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38002025-04-30T06:35:39.665ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38012025-04-30T06:35:39.665ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38022025-04-30T06:35:39.674ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38032025-04-30T06:35:39.679ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38042025-04-30T06:35:39.679ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
38052025-04-30T06:35:39.679ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38062025-04-30T06:35:39.679ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38072025-04-30T06:35:39.679ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38082025-04-30T06:35:39.680ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38092025-04-30T06:35:39.684ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
38102025-04-30T06:35:39.685ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38112025-04-30T06:35:39.688ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
38122025-04-30T06:35:39.688ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38132025-04-30T06:35:39.688ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38142025-04-30T06:35:39.688ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38152025-04-30T06:35:39.735ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38162025-04-30T06:35:39.735ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38172025-04-30T06:35:39.735ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38182025-04-30T06:35:39.735ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38192025-04-30T06:35:39.736ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38202025-04-30T06:35:39.744ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38212025-04-30T06:35:39.751ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38222025-04-30T06:35:39.751ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38232025-04-30T06:35:39.751ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38242025-04-30T06:35:39.752ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38252025-04-30T06:35:39.762ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38262025-04-30T06:35:39.762ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38272025-04-30T06:35:39.763ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38282025-04-30T06:35:39.844ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
38292025-04-30T06:35:39.844ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
38302025-04-30T06:35:39.845ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:54664 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
38312025-04-30T06:35:39.846ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:54664 req_id = 1afd4d35-8f3c-4b6a-b175-6feb234d2d11 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38322025-04-30T06:35:39.846ZINFOtest_metrics (dropshot): request completed latency_us = 563 local_addr = [::1]:56237 method = GET remote_addr = [::1]:54664 req_id = 1afd4d35-8f3c-4b6a-b175-6feb234d2d11 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
38332025-04-30T06:35:39.847ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "1afd4d35-8f3c-4b6a-b175-6feb234d2d11", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
38342025-04-30T06:35:39.848ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38352025-04-30T06:35:39.848ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
38362025-04-30T06:35:39.848ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
38372025-04-30T06:35:39.850ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38382025-04-30T06:35:39.850ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38392025-04-30T06:35:39.850ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38402025-04-30T06:35:39.850ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38412025-04-30T06:35:39.851ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38422025-04-30T06:35:39.861ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38432025-04-30T06:35:39.866ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38442025-04-30T06:35:39.867ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38452025-04-30T06:35:39.867ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38462025-04-30T06:35:39.868ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38472025-04-30T06:35:39.919ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
38482025-04-30T06:35:39.936ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
38492025-04-30T06:35:39.939ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
38502025-04-30T06:35:39.939ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.068563132s iteration = 5 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
38512025-04-30T06:35:39.939ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 6 reason = Signaled
38522025-04-30T06:35:39.939ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
38532025-04-30T06:35:39.939ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38542025-04-30T06:35:39.939ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38552025-04-30T06:35:39.939ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38562025-04-30T06:35:39.941ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38572025-04-30T06:35:39.957ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38582025-04-30T06:35:39.962ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
38592025-04-30T06:35:39.962ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
38602025-04-30T06:35:39.962ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
38612025-04-30T06:35:39.962ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
38622025-04-30T06:35:39.973ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
38632025-04-30T06:35:39.973ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
38642025-04-30T06:35:39.973ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38652025-04-30T06:35:39.973ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38662025-04-30T06:35:39.973ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38672025-04-30T06:35:39.974ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
38682025-04-30T06:35:39.974ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
38692025-04-30T06:35:39.974ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: da 98 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65061 req_id = e888d8ff-ccae-4a35-882a-21ef76030a3f
38702025-04-30T06:35:39.976ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55960,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65061 req_id = e888d8ff-ccae-4a35-882a-21ef76030a3f
38712025-04-30T06:35:39.977ZDEBGtest_metrics (store): query key key = _mgs._tcp
38722025-04-30T06:35:39.977ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38732025-04-30T06:35:39.977ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65061 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e888d8ff-ccae-4a35-882a-21ef76030a3f
38742025-04-30T06:35:39.978ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3804, tv_nsec: 777274988 } })
38752025-04-30T06:35:39.978ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
38762025-04-30T06:35:39.978ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ef 06 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52646 req_id = 4a35344e-b937-4530-a4ab-ae2482136ce8
38772025-04-30T06:35:39.978ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 61190,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52646 req_id = 4a35344e-b937-4530-a4ab-ae2482136ce8
38782025-04-30T06:35:39.979ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
38792025-04-30T06:35:39.979ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52646 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4a35344e-b937-4530-a4ab-ae2482136ce8
38802025-04-30T06:35:40.099ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
38812025-04-30T06:35:40.100ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
38822025-04-30T06:35:40.106ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
38832025-04-30T06:35:40.106ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38842025-04-30T06:35:40.106ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38852025-04-30T06:35:40.106ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38862025-04-30T06:35:40.107ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 21 88 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c !............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63665 req_id = 20ac4a42-3dcb-48d3-94a5-c15b63517ca3
38872025-04-30T06:35:40.107ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8584,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63665 req_id = 20ac4a42-3dcb-48d3-94a5-c15b63517ca3
38882025-04-30T06:35:40.107ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
38892025-04-30T06:35:40.107ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8584, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63665 req_id = 20ac4a42-3dcb-48d3-94a5-c15b63517ca3
38902025-04-30T06:35:40.108ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
38912025-04-30T06:35:40.108ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
38922025-04-30T06:35:40.108ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
38932025-04-30T06:35:40.111ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
38942025-04-30T06:35:40.111ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
38952025-04-30T06:35:40.112ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
38962025-04-30T06:35:40.113ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce9b4432-6079-4b6e-8b51-cbc97d80e0cd", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
38972025-04-30T06:35:40.113ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
38982025-04-30T06:35:40.117ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aabdc880-78f7-4aac-a364-9a9d928f3db9", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
38992025-04-30T06:35:40.117ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
39002025-04-30T06:35:40.119ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "815efaa6-cf51-4d05-9212-1cd899bdbdea", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39012025-04-30T06:35:40.119ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
39022025-04-30T06:35:40.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0650ac11-d8a2-4084-9e92-051509d4ebfe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39032025-04-30T06:35:40.122ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
39042025-04-30T06:35:40.124ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09e51fce-70bd-417a-aa04-52d2a13c9258", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39052025-04-30T06:35:40.124ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
39062025-04-30T06:35:40.127ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5da814c1-2867-4e6f-bbd7-406566d546c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39072025-04-30T06:35:40.128ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
39082025-04-30T06:35:40.131ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b9918b6-bdac-42d9-be38-40dd9c245280", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39092025-04-30T06:35:40.131ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
39102025-04-30T06:35:40.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bc0f8be-b140-4ab2-adcb-691bb52c34ab", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39112025-04-30T06:35:40.133ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
39122025-04-30T06:35:40.134ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c83566ef-4102-4712-82f5-e187ff5a8e7f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39132025-04-30T06:35:40.135ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
39142025-04-30T06:35:40.136ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d6c662c-e858-4423-97cf-2bcee7ee95f1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39152025-04-30T06:35:40.136ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
39162025-04-30T06:35:40.137ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0641a9fa-038f-460f-a7f2-d4eeb4908976", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39172025-04-30T06:35:40.137ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
39182025-04-30T06:35:40.140ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
39192025-04-30T06:35:40.141ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ec3652-8d1c-43d4-a887-a92466ecc79b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39202025-04-30T06:35:40.141ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
39212025-04-30T06:35:40.143ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97dee90e-4022-4018-a374-dac21b548758", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39222025-04-30T06:35:40.143ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
39232025-04-30T06:35:40.145ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e503f9f-941a-4a7a-a494-1acd08f81d0a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39242025-04-30T06:35:40.145ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
39252025-04-30T06:35:40.147ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17dec519-eff1-4a54-a4e4-fd3fc2be898d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39262025-04-30T06:35:40.147ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
39272025-04-30T06:35:40.150ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ed1c549-aec2-4e09-ad07-cb7d2666c6f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39282025-04-30T06:35:40.150ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
39292025-04-30T06:35:40.153ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "44ca4d21-ae7f-4cda-9c05-813ba1a3b5c5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39302025-04-30T06:35:40.154ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
39312025-04-30T06:35:40.157ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7be4fd0c-80a9-4e85-9aab-9c440eff444d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39322025-04-30T06:35:40.157ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
39332025-04-30T06:35:40.160ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "361b7f6c-21b6-43aa-90d9-ccb89b8368e3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39342025-04-30T06:35:40.160ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
39352025-04-30T06:35:40.162ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bef1d5fe-6242-4c8a-bbd8-5cfc18886200", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39362025-04-30T06:35:40.168ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
39372025-04-30T06:35:40.170ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bd1a93b-cc05-4861-a766-ce4aaabf5f0e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39382025-04-30T06:35:40.171ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
39392025-04-30T06:35:40.172ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "759b9e1a-4580-4da9-aeed-f691b53b9bb9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39402025-04-30T06:35:40.172ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
39412025-04-30T06:35:40.175ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "393c1586-667f-4b99-9c8d-5bf6d4e1be67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39422025-04-30T06:35:40.175ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
39432025-04-30T06:35:40.178ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dd52e41-b3be-4f6d-bab6-e0fc64da2b51", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39442025-04-30T06:35:40.178ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
39452025-04-30T06:35:40.182ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b829191-8f0e-46f9-ad68-6ef87e217170", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39462025-04-30T06:35:40.182ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
39472025-04-30T06:35:40.186ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "927cab26-1169-47d3-b716-50f763f93022", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39482025-04-30T06:35:40.186ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
39492025-04-30T06:35:40.188ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "407b8c58-b65d-4979-b0a3-bf1da4de0eb5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39502025-04-30T06:35:40.188ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
39512025-04-30T06:35:40.190ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62392c9b-18ac-4054-a938-0d34e5d30730", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39522025-04-30T06:35:40.190ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
39532025-04-30T06:35:40.191ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
39542025-04-30T06:35:40.191ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
39552025-04-30T06:35:40.191ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39562025-04-30T06:35:40.191ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39572025-04-30T06:35:40.191ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39582025-04-30T06:35:40.193ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ee29ea2-445c-494b-98a3-ef03f210767a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39592025-04-30T06:35:40.194ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
39602025-04-30T06:35:40.195ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
39612025-04-30T06:35:40.200ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
39622025-04-30T06:35:40.200ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39632025-04-30T06:35:40.200ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39642025-04-30T06:35:40.201ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39652025-04-30T06:35:40.202ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c2ed01-4efb-40e8-8ea7-d80d82fdfad9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39662025-04-30T06:35:40.203ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
39672025-04-30T06:35:40.205ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
39682025-04-30T06:35:40.205ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
39692025-04-30T06:35:40.206ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
39702025-04-30T06:35:40.208ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "799617ae-3c44-4530-b9ee-9a2bd24f830a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39712025-04-30T06:35:40.208ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
39722025-04-30T06:35:40.210ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37db0e00-0b43-4cd6-b600-f90cfc301122", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39732025-04-30T06:35:40.211ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
39742025-04-30T06:35:40.214ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80cadb9-e9f5-4189-98ca-6f98d91ca437", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39752025-04-30T06:35:40.214ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
39762025-04-30T06:35:40.217ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29a290f3-9e9c-4966-ac40-dd82c63079db", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39772025-04-30T06:35:40.217ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
39782025-04-30T06:35:40.230ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "58ad3098-c27f-4593-9b2d-13ffc76c956d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39792025-04-30T06:35:40.230ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
39802025-04-30T06:35:40.233ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e03c40e1-0d9b-462c-8596-a5ee54674a5b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39812025-04-30T06:35:40.233ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
39822025-04-30T06:35:40.240ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49e27bde-dfe4-4489-af11-dac0ac1146c8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39832025-04-30T06:35:40.240ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
39842025-04-30T06:35:40.249ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
39852025-04-30T06:35:40.251ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b46aadba-9926-462f-8e92-0c704e702155", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39862025-04-30T06:35:40.251ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
39872025-04-30T06:35:40.255ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a6c11a-5129-477a-8ca6-891a0a2b13f4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39882025-04-30T06:35:40.255ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
39892025-04-30T06:35:40.258ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "600c1d87-090a-44e0-acf0-9886ec58534f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39902025-04-30T06:35:40.258ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
39912025-04-30T06:35:40.265ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f7b6934-f2ff-4795-9d82-5e59d4924519", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39922025-04-30T06:35:40.265ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
39932025-04-30T06:35:40.271ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "faf8d762-c22b-4360-96ae-68a4b3a77446", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39942025-04-30T06:35:40.271ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
39952025-04-30T06:35:40.273ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "89cab799-5399-4855-ab2f-553d5efb5729", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39962025-04-30T06:35:40.273ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
39972025-04-30T06:35:40.275ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "063bdbb5-b42c-4947-a262-87851dc025dd", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
39982025-04-30T06:35:40.275ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
39992025-04-30T06:35:40.276ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3938a43-1baa-4b37-a395-3bf5d83fcf08", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:39 GMT"} })
40002025-04-30T06:35:40.277ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40012025-04-30T06:35:40.277ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40022025-04-30T06:35:40.277ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40032025-04-30T06:35:40.277ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40042025-04-30T06:35:40.278ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40052025-04-30T06:35:40.281ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40062025-04-30T06:35:40.294ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40072025-04-30T06:35:40.294ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
40082025-04-30T06:35:40.294ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40092025-04-30T06:35:40.294ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40102025-04-30T06:35:40.294ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40112025-04-30T06:35:40.295ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40122025-04-30T06:35:40.301ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40132025-04-30T06:35:40.302ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40142025-04-30T06:35:40.302ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40152025-04-30T06:35:40.302ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40162025-04-30T06:35:40.303ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40172025-04-30T06:35:40.303ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40182025-04-30T06:35:40.309ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
40192025-04-30T06:35:40.309ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40202025-04-30T06:35:40.309ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40212025-04-30T06:35:40.309ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40222025-04-30T06:35:40.311ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40232025-04-30T06:35:40.317ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
40242025-04-30T06:35:40.317ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40252025-04-30T06:35:40.317ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40262025-04-30T06:35:40.318ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40272025-04-30T06:35:40.386ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
40282025-04-30T06:35:40.386ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
40292025-04-30T06:35:40.389ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:59247 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
40302025-04-30T06:35:40.390ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40312025-04-30T06:35:40.390ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40322025-04-30T06:35:40.391ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40332025-04-30T06:35:40.393ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:59247 req_id = e455f397-5764-49a4-a334-6be9a3f25f9b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40342025-04-30T06:35:40.394ZINFOtest_metrics (dropshot): request completed latency_us = 717 local_addr = [::1]:56237 method = GET remote_addr = [::1]:59247 req_id = e455f397-5764-49a4-a334-6be9a3f25f9b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
40352025-04-30T06:35:40.395ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e455f397-5764-49a4-a334-6be9a3f25f9b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
40362025-04-30T06:35:40.398ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40372025-04-30T06:35:40.398ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
40382025-04-30T06:35:40.398ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
40392025-04-30T06:35:40.400ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40402025-04-30T06:35:40.400ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40412025-04-30T06:35:40.400ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40422025-04-30T06:35:40.400ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40432025-04-30T06:35:40.401ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40442025-04-30T06:35:40.415ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40452025-04-30T06:35:40.488ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40462025-04-30T06:35:40.488ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40472025-04-30T06:35:40.488ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40482025-04-30T06:35:40.489ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40492025-04-30T06:35:40.522ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40502025-04-30T06:35:40.522ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40512025-04-30T06:35:40.582ZTRACtest_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
40522025-04-30T06:35:40.582ZTRACtest_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
40532025-04-30T06:35:40.589ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40542025-04-30T06:35:40.590ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40552025-04-30T06:35:40.590ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40562025-04-30T06:35:40.590ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
40572025-04-30T06:35:40.663ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
40582025-04-30T06:35:40.665ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
40592025-04-30T06:35:40.665ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 725.919377ms iteration = 6 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
40602025-04-30T06:35:40.665ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 7 reason = Signaled
40612025-04-30T06:35:40.666ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
40622025-04-30T06:35:40.666ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40632025-04-30T06:35:40.666ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40642025-04-30T06:35:40.666ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40652025-04-30T06:35:40.667ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40662025-04-30T06:35:40.668ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
40672025-04-30T06:35:40.682ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40682025-04-30T06:35:40.693ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
40692025-04-30T06:35:40.693ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
40702025-04-30T06:35:40.693ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
40712025-04-30T06:35:40.694ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
40722025-04-30T06:35:40.708ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
40732025-04-30T06:35:40.708ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
40742025-04-30T06:35:40.713ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 60 ef 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 `............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50120 req_id = e9746298-cfff-4803-961e-14167416d38b
40752025-04-30T06:35:40.713ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24815,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50120 req_id = e9746298-cfff-4803-961e-14167416d38b
40762025-04-30T06:35:40.713ZDEBGtest_metrics (store): query key key = _mgs._tcp
40772025-04-30T06:35:40.713ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40782025-04-30T06:35:40.714ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50120 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e9746298-cfff-4803-961e-14167416d38b
40792025-04-30T06:35:40.714ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3805, tv_nsec: 513357334 } })
40802025-04-30T06:35:40.714ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
40812025-04-30T06:35:40.715ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2f 4d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e /M..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62254 req_id = 15cf0436-95fa-4a41-b689-83f418959e55
40822025-04-30T06:35:40.715ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12109,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62254 req_id = 15cf0436-95fa-4a41-b689-83f418959e55
40832025-04-30T06:35:40.715ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
40842025-04-30T06:35:40.715ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62254 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 15cf0436-95fa-4a41-b689-83f418959e55
40852025-04-30T06:35:40.774ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
40862025-04-30T06:35:40.775ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
40872025-04-30T06:35:40.775ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
40882025-04-30T06:35:40.775ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
40892025-04-30T06:35:40.775ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
40902025-04-30T06:35:40.776ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
40912025-04-30T06:35:40.777ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c2 05 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60753 req_id = 7480d01a-7e6c-4564-be8f-cd1ef70b60d6
40922025-04-30T06:35:40.777ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49669,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60753 req_id = 7480d01a-7e6c-4564-be8f-cd1ef70b60d6
40932025-04-30T06:35:40.777ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
40942025-04-30T06:35:40.777ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 49669, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60753 req_id = 7480d01a-7e6c-4564-be8f-cd1ef70b60d6
40952025-04-30T06:35:40.778ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
40962025-04-30T06:35:40.778ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
40972025-04-30T06:35:40.778ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
40982025-04-30T06:35:40.779ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
40992025-04-30T06:35:40.785ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41002025-04-30T06:35:40.786ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41012025-04-30T06:35:40.786ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41022025-04-30T06:35:40.786ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41032025-04-30T06:35:40.789ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "72c1b23a-22ae-44cf-ae8f-f256f573d3d7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41042025-04-30T06:35:40.789ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
41052025-04-30T06:35:40.790ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41062025-04-30T06:35:40.790ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41072025-04-30T06:35:40.790ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41082025-04-30T06:35:40.792ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d083ef13-15f9-4d57-8870-e7dc939dc4e1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41092025-04-30T06:35:40.793ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
41102025-04-30T06:35:40.798ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f332189-0e29-4c8f-81a5-4d0aa5aba8bd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41112025-04-30T06:35:40.798ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
41122025-04-30T06:35:40.800ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4491ba09-c028-4cb4-8c21-0c96fa2c7bfa", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41132025-04-30T06:35:40.800ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
41142025-04-30T06:35:40.805ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19c40eb6-d5eb-42dd-bb3b-bc684871f169", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41152025-04-30T06:35:40.805ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
41162025-04-30T06:35:40.808ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae491ed3-0deb-4ceb-959e-2b40e7b128a5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41172025-04-30T06:35:40.809ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
41182025-04-30T06:35:40.812ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ace4a8d1-1544-4471-85e8-e1bc8c600157", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41192025-04-30T06:35:40.812ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
41202025-04-30T06:35:40.814ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb005ce2-edec-4fa7-b68e-43a03269853f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41212025-04-30T06:35:40.815ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
41222025-04-30T06:35:40.820ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a240063d-551a-4ea6-bb0e-3fbfac41aebb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41232025-04-30T06:35:40.820ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
41242025-04-30T06:35:40.826ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc134766-60b3-45a0-8467-59fba6402723", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41252025-04-30T06:35:40.826ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
41262025-04-30T06:35:40.827ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
41272025-04-30T06:35:40.829ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3f8e2c8-f5f6-4dfb-902b-22156e36aa30", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41282025-04-30T06:35:40.829ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
41292025-04-30T06:35:40.831ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "891c91b3-43b2-4682-9e1f-217a18494200", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41302025-04-30T06:35:40.831ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
41312025-04-30T06:35:40.833ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9754b0fa-e6f2-46f5-9da6-0a3b7183caa5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41322025-04-30T06:35:40.833ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
41332025-04-30T06:35:40.836ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "53fe2b6d-7d90-4601-904d-6726eec34618", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41342025-04-30T06:35:40.836ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
41352025-04-30T06:35:40.839ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb714394-04ce-4eeb-844f-0876d97a7c73", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41362025-04-30T06:35:40.839ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
41372025-04-30T06:35:40.842ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c5e089-6fd5-4ce4-ba0c-fb401d33e342", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41382025-04-30T06:35:40.842ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
41392025-04-30T06:35:40.846ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4eb1dd54-fabf-4ce0-a93c-4d260b36479e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41402025-04-30T06:35:40.847ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
41412025-04-30T06:35:40.849ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfef06d5-2e68-47b4-bfd9-7e904c75b1a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41422025-04-30T06:35:40.849ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
41432025-04-30T06:35:40.851ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf16b9dc-07cb-4471-9a01-9565348e8c99", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41442025-04-30T06:35:40.851ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
41452025-04-30T06:35:40.852ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b829071-aeb3-4c1b-9887-7e515f9d30c1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41462025-04-30T06:35:40.852ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
41472025-04-30T06:35:40.853ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15d0b132-2f3c-4a15-8ecb-9781fa2d3449", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41482025-04-30T06:35:40.853ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
41492025-04-30T06:35:40.855ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee4ca92-81de-435f-9924-a9e86509d470", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41502025-04-30T06:35:40.855ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
41512025-04-30T06:35:40.856ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1a18c90-9dff-4606-a4b2-2af57aad124f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41522025-04-30T06:35:40.856ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
41532025-04-30T06:35:40.858ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7dbdbd2-c4a8-4491-93e4-239cecfecd24", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41542025-04-30T06:35:40.858ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
41552025-04-30T06:35:40.864ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2583ce1-f3c0-4349-be2b-686014b36ec0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41562025-04-30T06:35:40.864ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
41572025-04-30T06:35:40.869ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49571626-0b6e-40c6-ad3d-70734d86970b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41582025-04-30T06:35:40.869ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
41592025-04-30T06:35:40.871ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bfd9947-37a9-4236-8757-8188c1618ebf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41602025-04-30T06:35:40.871ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
41612025-04-30T06:35:40.875ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4de6689e-7c4c-406b-a228-ad756bc6910e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41622025-04-30T06:35:40.875ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
41632025-04-30T06:35:40.881ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
41642025-04-30T06:35:40.881ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
41652025-04-30T06:35:40.881ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41662025-04-30T06:35:40.881ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41672025-04-30T06:35:40.882ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41682025-04-30T06:35:40.882ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d55cf841-7ee2-4a56-93e3-2db74ddae1d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41692025-04-30T06:35:40.882ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
41702025-04-30T06:35:40.884ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
41712025-04-30T06:35:40.890ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
41722025-04-30T06:35:40.890ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41732025-04-30T06:35:40.890ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41742025-04-30T06:35:40.890ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41752025-04-30T06:35:40.893ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
41762025-04-30T06:35:40.893ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
41772025-04-30T06:35:40.893ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
41782025-04-30T06:35:40.895ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "025194eb-2d6c-4707-96a6-74a8bcd85a88", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41792025-04-30T06:35:40.896ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
41802025-04-30T06:35:40.897ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7802e77e-516a-4e9c-88dd-489e450e38e6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41812025-04-30T06:35:40.897ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
41822025-04-30T06:35:40.899ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e52102c-67b6-4e61-aa23-3c7c76948a7a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41832025-04-30T06:35:40.899ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
41842025-04-30T06:35:40.901ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a957c9cb-c39b-40cd-bb86-36dbea2b8c12", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41852025-04-30T06:35:40.902ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
41862025-04-30T06:35:40.907ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39cb7eda-1b05-4c3d-a808-b8824d71a144", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41872025-04-30T06:35:40.907ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
41882025-04-30T06:35:40.909ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97ed6f86-5ee7-466e-a232-31ad52d52221", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41892025-04-30T06:35:40.909ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
41902025-04-30T06:35:40.911ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc64652a-94c6-4375-aa15-2813708725c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41912025-04-30T06:35:40.911ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
41922025-04-30T06:35:40.915ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc056591-5c88-484f-813f-cb438e7e453b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41932025-04-30T06:35:40.923ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
41942025-04-30T06:35:40.926ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb5690b7-3200-4bb0-bcff-74f93129dd4c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41952025-04-30T06:35:40.926ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
41962025-04-30T06:35:40.929ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85bea6f0-3bd6-4177-9549-7c542434675e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41972025-04-30T06:35:40.929ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
41982025-04-30T06:35:40.931ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b45b3a-58f7-44b5-9c82-bcbd4bcfd225", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
41992025-04-30T06:35:40.931ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
42002025-04-30T06:35:40.934ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
42012025-04-30T06:35:40.936ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ec0377-5ac8-44c3-96af-a50ea10b27f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
42022025-04-30T06:35:40.936ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
42032025-04-30T06:35:40.939ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fb95ee0-8bd6-4aa2-b3af-18ebce6f6f76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
42042025-04-30T06:35:40.939ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
42052025-04-30T06:35:40.941ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "64a54e44-dfdf-41b1-94c3-5a36c51dffbb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
42062025-04-30T06:35:40.941ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
42072025-04-30T06:35:40.942ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8a01aa6-9aba-4e7f-aaaa-00fd9c53cbc3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
42082025-04-30T06:35:40.943ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
42092025-04-30T06:35:40.945ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c63a59-e90a-47b7-944d-f94d03c39e2d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
42102025-04-30T06:35:40.945ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42112025-04-30T06:35:40.945ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42122025-04-30T06:35:40.945ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42132025-04-30T06:35:40.945ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42142025-04-30T06:35:40.946ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42152025-04-30T06:35:40.963ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42162025-04-30T06:35:40.968ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42172025-04-30T06:35:40.968ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
42182025-04-30T06:35:40.968ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42192025-04-30T06:35:40.968ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42202025-04-30T06:35:40.968ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42212025-04-30T06:35:40.968ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42222025-04-30T06:35:40.973ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42232025-04-30T06:35:40.977ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
42242025-04-30T06:35:40.977ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42252025-04-30T06:35:40.977ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42262025-04-30T06:35:40.978ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42272025-04-30T06:35:41.118ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
42282025-04-30T06:35:41.118ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
42292025-04-30T06:35:41.119ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42302025-04-30T06:35:41.119ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42312025-04-30T06:35:41.119ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42322025-04-30T06:35:41.119ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42332025-04-30T06:35:41.120ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42342025-04-30T06:35:41.120ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:59719 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
42352025-04-30T06:35:41.122ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:59719 req_id = d49e59f4-104c-4ad4-a02c-098f784bb0df sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42362025-04-30T06:35:41.122ZINFOtest_metrics (dropshot): request completed latency_us = 781 local_addr = [::1]:56237 method = GET remote_addr = [::1]:59719 req_id = d49e59f4-104c-4ad4-a02c-098f784bb0df response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
42372025-04-30T06:35:41.123ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d49e59f4-104c-4ad4-a02c-098f784bb0df", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:40 GMT"} })
42382025-04-30T06:35:41.126ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42392025-04-30T06:35:41.126ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
42402025-04-30T06:35:41.126ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
42412025-04-30T06:35:41.129ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42422025-04-30T06:35:41.129ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42432025-04-30T06:35:41.129ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42442025-04-30T06:35:41.129ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42452025-04-30T06:35:41.130ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42462025-04-30T06:35:41.131ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42472025-04-30T06:35:41.137ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
42482025-04-30T06:35:41.137ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42492025-04-30T06:35:41.138ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42502025-04-30T06:35:41.138ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42512025-04-30T06:35:41.140ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42522025-04-30T06:35:41.165ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42532025-04-30T06:35:41.165ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42542025-04-30T06:35:41.166ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42552025-04-30T06:35:41.166ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42562025-04-30T06:35:41.171ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42572025-04-30T06:35:41.172ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42582025-04-30T06:35:41.172ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42592025-04-30T06:35:41.205ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
42602025-04-30T06:35:41.206ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
42612025-04-30T06:35:41.206ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 540.350743ms iteration = 7 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
42622025-04-30T06:35:41.206ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 8 reason = Signaled
42632025-04-30T06:35:41.206ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
42642025-04-30T06:35:41.206ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42652025-04-30T06:35:41.206ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42662025-04-30T06:35:41.206ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42672025-04-30T06:35:41.208ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42682025-04-30T06:35:41.220ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
42692025-04-30T06:35:41.221ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
42702025-04-30T06:35:41.230ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
42712025-04-30T06:35:41.231ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
42722025-04-30T06:35:41.231ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
42732025-04-30T06:35:41.233ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
42742025-04-30T06:35:41.239ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
42752025-04-30T06:35:41.239ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
42762025-04-30T06:35:41.239ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4e 53 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 NS..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59017 req_id = 1e989510-1f3a-42ad-8ed8-3f601819233e
42772025-04-30T06:35:41.239ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20051,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59017 req_id = 1e989510-1f3a-42ad-8ed8-3f601819233e
42782025-04-30T06:35:41.240ZDEBGtest_metrics (store): query key key = _mgs._tcp
42792025-04-30T06:35:41.240ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42802025-04-30T06:35:41.240ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59017 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1e989510-1f3a-42ad-8ed8-3f601819233e
42812025-04-30T06:35:41.241ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3806, tv_nsec: 40006207 } })
42822025-04-30T06:35:41.241ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
42832025-04-30T06:35:41.241ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: df 98 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59799 req_id = 5c9febb2-cf8a-4f78-ae2d-6d1b942de335
42842025-04-30T06:35:41.241ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57240,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59799 req_id = 5c9febb2-cf8a-4f78-ae2d-6d1b942de335
42852025-04-30T06:35:41.241ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
42862025-04-30T06:35:41.241ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59799 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5c9febb2-cf8a-4f78-ae2d-6d1b942de335
42872025-04-30T06:35:41.376ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
42882025-04-30T06:35:41.377ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
42892025-04-30T06:35:41.377ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
42902025-04-30T06:35:41.382ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
42912025-04-30T06:35:41.382ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
42922025-04-30T06:35:41.384ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
42932025-04-30T06:35:41.384ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
42942025-04-30T06:35:41.387ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
42952025-04-30T06:35:41.389ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 15cee702-78a9-4d45-b6b2-b90a5ee70946 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
42962025-04-30T06:35:41.389ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e8 b3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51512 req_id = b71c0dd4-b791-431c-b4c8-19454c29c78e
42972025-04-30T06:35:41.390ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59571,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51512 req_id = b71c0dd4-b791-431c-b4c8-19454c29c78e
42982025-04-30T06:35:41.390ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
42992025-04-30T06:35:41.390ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59571, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51512 req_id = b71c0dd4-b791-431c-b4c8-19454c29c78e
43002025-04-30T06:35:41.390ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 15cee702-78a9-4d45-b6b2-b90a5ee70946 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43012025-04-30T06:35:41.390ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 15cee702-78a9-4d45-b6b2-b90a5ee70946 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43022025-04-30T06:35:41.391ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 15cee702-78a9-4d45-b6b2-b90a5ee70946 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43032025-04-30T06:35:41.392ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43042025-04-30T06:35:41.396ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43052025-04-30T06:35:41.442ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43062025-04-30T06:35:41.443ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43072025-04-30T06:35:41.444ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43082025-04-30T06:35:41.445ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
43092025-04-30T06:35:41.445ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
43102025-04-30T06:35:41.445ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
43112025-04-30T06:35:41.450ZINFOtest_metrics (dropshot_internal): request completed latency_us = 61044 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 15cee702-78a9-4d45-b6b2-b90a5ee70946 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
43122025-04-30T06:35:41.451ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43132025-04-30T06:35:41.451ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43142025-04-30T06:35:41.451ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43152025-04-30T06:35:41.452ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "15cee702-78a9-4d45-b6b2-b90a5ee70946", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43162025-04-30T06:35:41.453ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43172025-04-30T06:35:41.461ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5bfe80b5-d0fa-4511-956e-4aa83e04e57a uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43182025-04-30T06:35:41.464ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5bfe80b5-d0fa-4511-956e-4aa83e04e57a resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43192025-04-30T06:35:41.464ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5bfe80b5-d0fa-4511-956e-4aa83e04e57a roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43202025-04-30T06:35:41.464ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5bfe80b5-d0fa-4511-956e-4aa83e04e57a resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43212025-04-30T06:35:41.481ZINFOtest_metrics (dropshot_internal): request completed latency_us = 20608 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5bfe80b5-d0fa-4511-956e-4aa83e04e57a response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
43222025-04-30T06:35:41.484ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bfe80b5-d0fa-4511-956e-4aa83e04e57a", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43232025-04-30T06:35:41.484ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
43242025-04-30T06:35:41.484ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43252025-04-30T06:35:41.485ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
43262025-04-30T06:35:41.485ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b898156b-0b42-493a-adba-11dc03c2cd66", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43272025-04-30T06:35:41.485ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
43282025-04-30T06:35:41.508ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21094603-d28c-4f6d-b78d-5c6c6753798c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43292025-04-30T06:35:41.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
43302025-04-30T06:35:41.515ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43312025-04-30T06:35:41.515ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43322025-04-30T06:35:41.521ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43332025-04-30T06:35:41.527ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43342025-04-30T06:35:41.527ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43352025-04-30T06:35:41.527ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
43362025-04-30T06:35:41.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d65785f3-2e09-4fdb-b9c4-811f43d09d74", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43372025-04-30T06:35:41.532ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
43382025-04-30T06:35:41.534ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf304539-f65c-4e03-9325-1b1257762c81", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43392025-04-30T06:35:41.534ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
43402025-04-30T06:35:41.536ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f9a3234-7014-453d-8c66-87ce931ae93f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43412025-04-30T06:35:41.536ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
43422025-04-30T06:35:41.538ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c31a35aa-be86-45b6-91b1-681a0148b16c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43432025-04-30T06:35:41.538ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
43442025-04-30T06:35:41.542ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "44ffdacf-05b0-47d7-9f3c-347c29ca1ab8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43452025-04-30T06:35:41.542ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
43462025-04-30T06:35:41.543ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
43472025-04-30T06:35:41.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48a48bb6-4f91-4b7c-9c16-3fee26a66716", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43482025-04-30T06:35:41.583ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
43492025-04-30T06:35:41.587ZDEBGtest_metrics (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43502025-04-30T06:35:41.587ZDEBGtest_metrics (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43512025-04-30T06:35:41.588ZDEBGtest_metrics (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
43522025-04-30T06:35:41.589ZDEBGtest_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
43532025-04-30T06:35:41.589ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT * FROM oximeter.timeseries_schema FORMAT Native;
43542025-04-30T06:35:41.590ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30d525ab-f42d-4e26-8952-e78415d3a9c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43552025-04-30T06:35:41.590ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
43562025-04-30T06:35:41.592ZTRACtest_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
43572025-04-30T06:35:41.593ZDEBGtest_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e schema = TimeseriesSchema { timeseries_name: TimeseriesName("integration_target:integration_metric"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "metric_name", field_type: String, source: Metric, description: "" }, FieldSchema { name: "target_name", field_type: String, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:35:41.589632056Z } timeseries_name = integration_target:integration_metric
43582025-04-30T06:35:41.594ZDEBGtest_metrics (clickhouse-client): inserting 1 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
43592025-04-30T06:35:41.595ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 6 n_rows = 1 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
43602025-04-30T06:35:41.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad495d4c-6ab6-4219-b8c1-68966315c4c0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43612025-04-30T06:35:41.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
43622025-04-30T06:35:41.596ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
43632025-04-30T06:35:41.596ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
43642025-04-30T06:35:41.596ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43652025-04-30T06:35:41.596ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43662025-04-30T06:35:41.598ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43672025-04-30T06:35:41.600ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d915d3ee-9e23-4cf4-92b7-0773da729cf7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43682025-04-30T06:35:41.600ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
43692025-04-30T06:35:41.602ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
43702025-04-30T06:35:41.608ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
43712025-04-30T06:35:41.608ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43722025-04-30T06:35:41.608ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43732025-04-30T06:35:41.609ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43742025-04-30T06:35:41.609ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
43752025-04-30T06:35:41.616ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba97ec31-c502-4e92-b46e-42e3c8e6bad5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43762025-04-30T06:35:41.617ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
43772025-04-30T06:35:41.617ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
43782025-04-30T06:35:41.617ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
43792025-04-30T06:35:41.618ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
43802025-04-30T06:35:41.624ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
43812025-04-30T06:35:41.624ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
43822025-04-30T06:35:41.629ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = measurements_i64
43832025-04-30T06:35:41.629ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
43842025-04-30T06:35:41.631ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef6f119c-1487-43c9-9bdb-87bac71dd766", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43852025-04-30T06:35:41.631ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
43862025-04-30T06:35:41.636ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6c839bd-25d0-44a9-bd1a-f8a9768a9a0c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43872025-04-30T06:35:41.636ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
43882025-04-30T06:35:41.640ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d4d2d68-6e42-40e3-89f7-5ebffe1f13db", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43892025-04-30T06:35:41.640ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
43902025-04-30T06:35:41.643ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee757753-d5c7-4161-85db-7096de404223", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43912025-04-30T06:35:41.643ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
43922025-04-30T06:35:41.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0655cf5-44fe-4659-9a82-5a8705ec70f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43932025-04-30T06:35:41.646ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
43942025-04-30T06:35:41.651ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3828d1-c9d3-468b-8126-862f2a67b7e8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43952025-04-30T06:35:41.651ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
43962025-04-30T06:35:41.657ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
43972025-04-30T06:35:41.657ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1386fa68-f1c6-4402-972e-e9575ee1bff7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
43982025-04-30T06:35:41.658ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
43992025-04-30T06:35:41.660ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f02e5e2c-d247-4e12-ac2a-3026c6d19887", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44002025-04-30T06:35:41.660ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
44012025-04-30T06:35:41.662ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb06b2e7-0d1e-48a8-8536-d9ab5a9770ca", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44022025-04-30T06:35:41.662ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
44032025-04-30T06:35:41.664ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e51b1715-9fe6-4ddf-901a-97bd346b06b8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44042025-04-30T06:35:41.665ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
44052025-04-30T06:35:41.666ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a613f81-061d-4b6b-a07f-de291f97f09a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44062025-04-30T06:35:41.667ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
44072025-04-30T06:35:41.669ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eda3dc5b-9fab-4459-965b-a96dd650e478", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44082025-04-30T06:35:41.669ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
44092025-04-30T06:35:41.672ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b682dfe2-bbe8-4941-aa03-425bb7c88773", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44102025-04-30T06:35:41.672ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
44112025-04-30T06:35:41.678ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1dd8ccce-e2be-4d4d-b75e-a284e521106f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44122025-04-30T06:35:41.678ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
44132025-04-30T06:35:41.680ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7a6a30-d882-4128-85c2-8369d72519fd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44142025-04-30T06:35:41.681ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
44152025-04-30T06:35:41.682ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c3e52be1-7bfb-49bf-9efe-0ab22c878007", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44162025-04-30T06:35:41.682ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
44172025-04-30T06:35:41.684ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b13ff208-589c-4017-bb4d-1deeaacd671c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44182025-04-30T06:35:41.684ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
44192025-04-30T06:35:41.687ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "38f43590-8602-407f-b709-aa91fa849e75", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44202025-04-30T06:35:41.688ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
44212025-04-30T06:35:41.689ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d67e144-b15c-44c5-a0f1-774b9bbdcba6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44222025-04-30T06:35:41.689ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
44232025-04-30T06:35:41.690ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
44242025-04-30T06:35:41.690ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e569e559-fb17-46fe-88ac-e5b7f6b49c52", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44252025-04-30T06:35:41.691ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
44262025-04-30T06:35:41.692ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a360016e-fb25-4342-9733-a996901ed8ec", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44272025-04-30T06:35:41.692ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
44282025-04-30T06:35:41.693ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "517da2d6-7c11-4bb6-87f8-794ed5011687", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44292025-04-30T06:35:41.693ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
44302025-04-30T06:35:41.694ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd337ee9-075b-44e5-8bfa-8eddaa9abfa4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44312025-04-30T06:35:41.695ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
44322025-04-30T06:35:41.697ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "31534522-59b2-4994-952a-207644c491bd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44332025-04-30T06:35:41.697ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
44342025-04-30T06:35:41.698ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "895f87a2-ed4c-49cb-be59-3fe7bf5bf114", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44352025-04-30T06:35:41.699ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
44362025-04-30T06:35:41.701ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88e762f9-11e0-4152-8bf6-838100a05681", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44372025-04-30T06:35:41.701ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
44382025-04-30T06:35:41.703ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3c4523d-c9f5-4732-aa27-ed37ec487940", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44392025-04-30T06:35:41.703ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
44402025-04-30T06:35:41.706ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b5107f3-0b5a-40e6-89aa-2c3d26d109a0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44412025-04-30T06:35:41.707ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
44422025-04-30T06:35:41.708ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
44432025-04-30T06:35:41.708ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44442025-04-30T06:35:41.708ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44452025-04-30T06:35:41.708ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
44462025-04-30T06:35:41.709ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44472025-04-30T06:35:41.711ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c9ee124-3774-4730-92cb-16bd59df8863", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44482025-04-30T06:35:41.711ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
44492025-04-30T06:35:41.712ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4b6e7fd-401b-4cee-8a28-0598018130d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44502025-04-30T06:35:41.712ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
44512025-04-30T06:35:41.713ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44522025-04-30T06:35:41.721ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
44532025-04-30T06:35:41.721ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44542025-04-30T06:35:41.721ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
44552025-04-30T06:35:41.721ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44562025-04-30T06:35:41.723ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d737472-b0d7-4b92-a80d-ca071b11e736", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44572025-04-30T06:35:41.724ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
44582025-04-30T06:35:41.724ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
44592025-04-30T06:35:41.724ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
44602025-04-30T06:35:41.724ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
44612025-04-30T06:35:41.725ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa61fb7d-92d3-43f7-b2da-eb460534dc20", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44622025-04-30T06:35:41.726ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
44632025-04-30T06:35:41.727ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b2e6cfd-23b5-4aef-a31c-54d4f9ff07de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44642025-04-30T06:35:41.727ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44652025-04-30T06:35:41.728ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44662025-04-30T06:35:41.728ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44672025-04-30T06:35:41.728ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44682025-04-30T06:35:41.728ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44692025-04-30T06:35:41.735ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44702025-04-30T06:35:41.744ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44712025-04-30T06:35:41.744ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
44722025-04-30T06:35:41.744ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44732025-04-30T06:35:41.744ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44742025-04-30T06:35:41.744ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44752025-04-30T06:35:41.745ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44762025-04-30T06:35:41.750ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44772025-04-30T06:35:41.756ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
44782025-04-30T06:35:41.756ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44792025-04-30T06:35:41.756ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44802025-04-30T06:35:41.756ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44812025-04-30T06:35:41.828ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
44822025-04-30T06:35:41.828ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
44832025-04-30T06:35:41.828ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:57765 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
44842025-04-30T06:35:41.829ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:57765 req_id = 5853eb4e-156d-4b36-8ef3-2cb10f7911ea sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44852025-04-30T06:35:41.830ZINFOtest_metrics (dropshot): request completed latency_us = 541 local_addr = [::1]:56237 method = GET remote_addr = [::1]:57765 req_id = 5853eb4e-156d-4b36-8ef3-2cb10f7911ea response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
44862025-04-30T06:35:41.830ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5853eb4e-156d-4b36-8ef3-2cb10f7911ea", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
44872025-04-30T06:35:41.833ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44882025-04-30T06:35:41.833ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
44892025-04-30T06:35:41.833ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
44902025-04-30T06:35:41.836ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
44912025-04-30T06:35:41.836ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
44922025-04-30T06:35:41.836ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44932025-04-30T06:35:41.836ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44942025-04-30T06:35:41.839ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
44952025-04-30T06:35:41.842ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
44962025-04-30T06:35:41.858ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
44972025-04-30T06:35:41.858ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
44982025-04-30T06:35:41.858ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
44992025-04-30T06:35:41.859ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45002025-04-30T06:35:41.871ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
45012025-04-30T06:35:41.912ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
45022025-04-30T06:35:41.912ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
45032025-04-30T06:35:41.912ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 706.216611ms iteration = 8 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
45042025-04-30T06:35:41.912ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 9 reason = Signaled
45052025-04-30T06:35:41.912ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
45062025-04-30T06:35:41.912ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45072025-04-30T06:35:41.912ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45082025-04-30T06:35:41.912ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45092025-04-30T06:35:41.914ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45102025-04-30T06:35:41.920ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45112025-04-30T06:35:41.925ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
45122025-04-30T06:35:41.925ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
45132025-04-30T06:35:41.925ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
45142025-04-30T06:35:41.926ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
45152025-04-30T06:35:41.927ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45162025-04-30T06:35:41.927ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45172025-04-30T06:35:41.927ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45182025-04-30T06:35:41.927ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45192025-04-30T06:35:41.927ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45202025-04-30T06:35:41.931ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
45212025-04-30T06:35:41.931ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
45222025-04-30T06:35:41.933ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 59 2a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 Y*..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63159 req_id = e1335768-8dbb-4e28-ab04-aa2627527a2d
45232025-04-30T06:35:41.934ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22826,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63159 req_id = e1335768-8dbb-4e28-ab04-aa2627527a2d
45242025-04-30T06:35:41.934ZDEBGtest_metrics (store): query key key = _mgs._tcp
45252025-04-30T06:35:41.934ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45262025-04-30T06:35:41.934ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63159 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e1335768-8dbb-4e28-ab04-aa2627527a2d
45272025-04-30T06:35:41.934ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45282025-04-30T06:35:41.943ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45292025-04-30T06:35:41.943ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45302025-04-30T06:35:41.943ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45312025-04-30T06:35:41.944ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45322025-04-30T06:35:41.944ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3806, tv_nsec: 743349913 } })
45332025-04-30T06:35:41.944ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
45342025-04-30T06:35:41.946ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: db 9b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59457 req_id = 22e70982-01ba-43e7-9a4b-61c7343f7ca0
45352025-04-30T06:35:41.946ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56219,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59457 req_id = 22e70982-01ba-43e7-9a4b-61c7343f7ca0
45362025-04-30T06:35:41.946ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
45372025-04-30T06:35:41.946ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59457 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 22e70982-01ba-43e7-9a4b-61c7343f7ca0
45382025-04-30T06:35:41.999ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
45392025-04-30T06:35:42.000ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45402025-04-30T06:35:42.000ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45412025-04-30T06:35:42.002ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45422025-04-30T06:35:42.002ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8f cb 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51485 req_id = 6ccb131f-f9bc-4491-94cc-e33f1134ef71
45432025-04-30T06:35:42.003ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36811,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51485 req_id = 6ccb131f-f9bc-4491-94cc-e33f1134ef71
45442025-04-30T06:35:42.003ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
45452025-04-30T06:35:42.003ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36811, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51485 req_id = 6ccb131f-f9bc-4491-94cc-e33f1134ef71
45462025-04-30T06:35:42.004ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
45472025-04-30T06:35:42.004ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
45482025-04-30T06:35:42.004ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
45492025-04-30T06:35:42.009ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d71a30-3b55-4a20-8499-ac354983cf9e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45502025-04-30T06:35:42.010ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
45512025-04-30T06:35:42.012ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a800424-4b99-44ac-90ee-ace16a594989", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45522025-04-30T06:35:42.012ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
45532025-04-30T06:35:42.017ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82737523-54ac-45d9-aada-5ff29737d731", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45542025-04-30T06:35:42.017ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
45552025-04-30T06:35:42.025ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaaf6f3e-270f-483e-9f57-bd36ed1bef4b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45562025-04-30T06:35:42.025ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
45572025-04-30T06:35:42.031ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f67d4896-c2d9-4511-ae9e-8993afe74d81", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45582025-04-30T06:35:42.031ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
45592025-04-30T06:35:42.038ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13611fa6-f960-4209-9a27-b9d31137fde4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45602025-04-30T06:35:42.038ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
45612025-04-30T06:35:42.040ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
45622025-04-30T06:35:42.046ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0699dc69-c0d3-4352-8640-eb96b604dd2a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45632025-04-30T06:35:42.046ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
45642025-04-30T06:35:42.048ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2149213d-1437-44e5-aa70-69f5d915ca4e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45652025-04-30T06:35:42.049ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
45662025-04-30T06:35:42.050ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "751cf6a9-32a5-4f2f-8fbe-48c0e273ec1f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45672025-04-30T06:35:42.050ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
45682025-04-30T06:35:42.052ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9b27416-0343-470e-804f-27c4cf612567", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45692025-04-30T06:35:42.052ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
45702025-04-30T06:35:42.053ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "098f1c66-7cc2-4c1a-a02c-d64aa94890e8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45712025-04-30T06:35:42.053ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
45722025-04-30T06:35:42.055ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44f83a74-6fb4-4eea-8ab3-cc5e693f2a14", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45732025-04-30T06:35:42.055ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
45742025-04-30T06:35:42.057ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43a5d4be-48ac-4c4c-9c3c-693c32cfb9cb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45752025-04-30T06:35:42.057ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
45762025-04-30T06:35:42.063ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c3c7d1c-c5af-468a-a068-9f289abb0a24", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45772025-04-30T06:35:42.063ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
45782025-04-30T06:35:42.068ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "80c5f30b-e33b-4a95-832c-ce33747f4150", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45792025-04-30T06:35:42.069ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
45802025-04-30T06:35:42.073ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9560f7a6-3deb-4c1d-bbca-28cc7e85146c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45812025-04-30T06:35:42.073ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
45822025-04-30T06:35:42.080ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3560a0d-22b9-4959-8238-1e4ad81e444a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45832025-04-30T06:35:42.083ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
45842025-04-30T06:35:42.095ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
45852025-04-30T06:35:42.095ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
45862025-04-30T06:35:42.095ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45872025-04-30T06:35:42.095ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45882025-04-30T06:35:42.098ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45892025-04-30T06:35:42.101ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
45902025-04-30T06:35:42.111ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
45912025-04-30T06:35:42.111ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45922025-04-30T06:35:42.111ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45932025-04-30T06:35:42.111ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45942025-04-30T06:35:42.115ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
45952025-04-30T06:35:42.115ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
45962025-04-30T06:35:42.116ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
45972025-04-30T06:35:42.119ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75385003-0bfb-4bdd-87af-0018ec717fc7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
45982025-04-30T06:35:42.119ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
45992025-04-30T06:35:42.124ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "886b6fe9-e1d2-4e7a-9a50-a61e51ee9551", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46002025-04-30T06:35:42.124ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
46012025-04-30T06:35:42.125ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f481ebf4-44ae-4764-a51b-20a6814ec601", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46022025-04-30T06:35:42.126ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
46032025-04-30T06:35:42.129ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b792482-0715-4b79-87ec-92fa9e092745", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46042025-04-30T06:35:42.129ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
46052025-04-30T06:35:42.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f195849-88f7-4b37-878b-02461c80d943", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46062025-04-30T06:35:42.133ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
46072025-04-30T06:35:42.136ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8e7328e-b459-4494-afbb-ec17c053dbc7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46082025-04-30T06:35:42.136ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
46092025-04-30T06:35:42.138ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e362d45-463d-4772-8d84-c360ab49288a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46102025-04-30T06:35:42.139ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
46112025-04-30T06:35:42.142ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9adbb782-1ab6-4b9d-8c48-fe4d9c584803", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46122025-04-30T06:35:42.143ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
46132025-04-30T06:35:42.149ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
46142025-04-30T06:35:42.149ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d0e1bf2-12b7-4a5a-908c-a848ef31be51", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46152025-04-30T06:35:42.149ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
46162025-04-30T06:35:42.154ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebcf9662-e2dd-4328-8116-1d11b2eb22e6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46172025-04-30T06:35:42.154ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
46182025-04-30T06:35:42.158ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e241adc5-8242-4992-8cb1-4fae72e3625c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46192025-04-30T06:35:42.159ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
46202025-04-30T06:35:42.162ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "517f5d29-fcb0-4391-8f18-d7abe58ffe6f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46212025-04-30T06:35:42.162ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
46222025-04-30T06:35:42.167ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "878fb204-8f2b-4406-a4f6-f6395de290e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46232025-04-30T06:35:42.167ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
46242025-04-30T06:35:42.169ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa857dbc-5b56-462d-a183-3fcce840c69b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46252025-04-30T06:35:42.169ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
46262025-04-30T06:35:42.170ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e69abb1-e4df-4f02-abf1-fbcbb565fd7d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46272025-04-30T06:35:42.170ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
46282025-04-30T06:35:42.172ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b26931a4-f6ce-4b23-bb16-edcc3f0095de", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46292025-04-30T06:35:42.173ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
46302025-04-30T06:35:42.175ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "87cfddb5-4949-4f45-b9a3-fe788e75ef30", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46312025-04-30T06:35:42.175ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
46322025-04-30T06:35:42.177ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1b14158-83c6-4667-a6fb-1fed4b52d540", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46332025-04-30T06:35:42.177ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
46342025-04-30T06:35:42.181ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afb0ef87-9432-41e3-8d23-e699d0c6835e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46352025-04-30T06:35:42.181ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
46362025-04-30T06:35:42.183ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0f0c8e4-fc74-418e-8959-ce7018a50aae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46372025-04-30T06:35:42.183ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
46382025-04-30T06:35:42.186ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1b5d93c-0ca6-4c35-aa97-091dacb699f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46392025-04-30T06:35:42.186ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
46402025-04-30T06:35:42.190ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f794fb8d-80e9-4947-882d-64bd5785dcdf", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46412025-04-30T06:35:42.191ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
46422025-04-30T06:35:42.194ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c2abbd7-4570-4353-9851-b987ae2ce29f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46432025-04-30T06:35:42.194ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
46442025-04-30T06:35:42.197ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18e001cc-39e8-4098-99f7-b55a20e6d44c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46452025-04-30T06:35:42.197ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
46462025-04-30T06:35:42.199ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1266f970-d945-45a4-a584-1e6db0c3c1da", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46472025-04-30T06:35:42.199ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
46482025-04-30T06:35:42.200ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
46492025-04-30T06:35:42.200ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46502025-04-30T06:35:42.200ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46512025-04-30T06:35:42.200ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46522025-04-30T06:35:42.201ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46532025-04-30T06:35:42.204ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6ffa628-8787-4248-bb82-9a8fe5d76961", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46542025-04-30T06:35:42.206ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
46552025-04-30T06:35:42.207ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d97fa3a4-a6a8-4a20-bcdd-4fa17859e688", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46562025-04-30T06:35:42.208ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
46572025-04-30T06:35:42.208ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46582025-04-30T06:35:42.216ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
46592025-04-30T06:35:42.216ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46602025-04-30T06:35:42.216ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46612025-04-30T06:35:42.217ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46622025-04-30T06:35:42.219ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "579929f0-1c55-46b6-9718-9f45a8414018", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46632025-04-30T06:35:42.219ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46642025-04-30T06:35:42.219ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46652025-04-30T06:35:42.219ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46662025-04-30T06:35:42.219ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46672025-04-30T06:35:42.219ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46682025-04-30T06:35:42.221ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
46692025-04-30T06:35:42.221ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
46702025-04-30T06:35:42.223ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
46712025-04-30T06:35:42.223ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46722025-04-30T06:35:42.228ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46732025-04-30T06:35:42.228ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
46742025-04-30T06:35:42.228ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46752025-04-30T06:35:42.228ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46762025-04-30T06:35:42.228ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46772025-04-30T06:35:42.229ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46782025-04-30T06:35:42.233ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46792025-04-30T06:35:42.238ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
46802025-04-30T06:35:42.238ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46812025-04-30T06:35:42.238ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46822025-04-30T06:35:42.240ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46832025-04-30T06:35:42.303ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
46842025-04-30T06:35:42.303ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
46852025-04-30T06:35:42.304ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:54679 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
46862025-04-30T06:35:42.305ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:54679 req_id = 5b0f9061-7c32-4e5c-b780-1d5be8c774ca sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46872025-04-30T06:35:42.305ZINFOtest_metrics (dropshot): request completed latency_us = 689 local_addr = [::1]:56237 method = GET remote_addr = [::1]:54679 req_id = 5b0f9061-7c32-4e5c-b780-1d5be8c774ca response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
46882025-04-30T06:35:42.306ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b0f9061-7c32-4e5c-b780-1d5be8c774ca", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:41 GMT"} })
46892025-04-30T06:35:42.309ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46902025-04-30T06:35:42.309ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
46912025-04-30T06:35:42.309ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
46922025-04-30T06:35:42.313ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
46932025-04-30T06:35:42.313ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
46942025-04-30T06:35:42.313ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
46952025-04-30T06:35:42.313ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
46962025-04-30T06:35:42.314ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
46972025-04-30T06:35:42.319ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
46982025-04-30T06:35:42.330ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
46992025-04-30T06:35:42.331ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47002025-04-30T06:35:42.331ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47012025-04-30T06:35:42.331ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47022025-04-30T06:35:42.367ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
47032025-04-30T06:35:42.405ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
47042025-04-30T06:35:42.405ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
47052025-04-30T06:35:42.405ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 492.746556ms iteration = 9 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
47062025-04-30T06:35:42.405ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 10 reason = Signaled
47072025-04-30T06:35:42.405ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
47082025-04-30T06:35:42.406ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47092025-04-30T06:35:42.406ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47102025-04-30T06:35:42.406ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47112025-04-30T06:35:42.407ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47122025-04-30T06:35:42.410ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47132025-04-30T06:35:42.416ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
47142025-04-30T06:35:42.416ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
47152025-04-30T06:35:42.417ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
47162025-04-30T06:35:42.417ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
47172025-04-30T06:35:42.418ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
47182025-04-30T06:35:42.418ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
47192025-04-30T06:35:42.418ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47202025-04-30T06:35:42.418ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
47212025-04-30T06:35:42.419ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47222025-04-30T06:35:42.421ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
47232025-04-30T06:35:42.421ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
47242025-04-30T06:35:42.423ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
47252025-04-30T06:35:42.430ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
47262025-04-30T06:35:42.430ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47272025-04-30T06:35:42.430ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
47282025-04-30T06:35:42.431ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47292025-04-30T06:35:42.431ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 03 d6 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52141 req_id = fdec34d2-58e1-40d1-a413-dca1f1247e90
47302025-04-30T06:35:42.431ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 982,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52141 req_id = fdec34d2-58e1-40d1-a413-dca1f1247e90
47312025-04-30T06:35:42.431ZDEBGtest_metrics (store): query key key = _mgs._tcp
47322025-04-30T06:35:42.432ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47332025-04-30T06:35:42.432ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52141 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = fdec34d2-58e1-40d1-a413-dca1f1247e90
47342025-04-30T06:35:42.434ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3807, tv_nsec: 233095470 } })
47352025-04-30T06:35:42.434ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
47362025-04-30T06:35:42.434ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 47 21 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e G!..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53104 req_id = 6bb4db90-fb14-4a0b-a9e1-dfab240f68a8
47372025-04-30T06:35:42.435ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 18209,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53104 req_id = 6bb4db90-fb14-4a0b-a9e1-dfab240f68a8
47382025-04-30T06:35:42.435ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
47392025-04-30T06:35:42.435ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53104 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6bb4db90-fb14-4a0b-a9e1-dfab240f68a8
47402025-04-30T06:35:42.435ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
47412025-04-30T06:35:42.435ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
47422025-04-30T06:35:42.442ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
47432025-04-30T06:35:42.505ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
47442025-04-30T06:35:42.506ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7c aa 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c |............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51590 req_id = 9ee97ceb-29b0-4eb9-978d-297b678e28ab
47452025-04-30T06:35:42.506ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31914,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51590 req_id = 9ee97ceb-29b0-4eb9-978d-297b678e28ab
47462025-04-30T06:35:42.507ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
47472025-04-30T06:35:42.507ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 31914, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51590 req_id = 9ee97ceb-29b0-4eb9-978d-297b678e28ab
47482025-04-30T06:35:42.507ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
47492025-04-30T06:35:42.507ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
47502025-04-30T06:35:42.507ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
47512025-04-30T06:35:42.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "43496309-c5f6-401f-8ca3-714402468073", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47522025-04-30T06:35:42.510ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
47532025-04-30T06:35:42.534ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47542025-04-30T06:35:42.534ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47552025-04-30T06:35:42.537ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47562025-04-30T06:35:42.538ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47572025-04-30T06:35:42.539ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47582025-04-30T06:35:42.539ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
47592025-04-30T06:35:42.539ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "684b0de3-1b20-48c4-adf3-a66b0b7af8b1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47602025-04-30T06:35:42.539ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
47612025-04-30T06:35:42.542ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e6b2d13-6eec-4ea6-a78a-37a7f048a690", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47622025-04-30T06:35:42.542ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
47632025-04-30T06:35:42.545ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f3b2962-8e5f-48ff-b2c9-50ceb5f9a6e7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47642025-04-30T06:35:42.545ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
47652025-04-30T06:35:42.550ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "291e058e-cb79-460e-bcc2-13fe467725e5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47662025-04-30T06:35:42.551ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
47672025-04-30T06:35:42.555ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb34b8e0-b864-4c19-824d-7419638a8fad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47682025-04-30T06:35:42.555ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
47692025-04-30T06:35:42.560ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c626278-103f-4aff-8871-1848a0af9cb3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47702025-04-30T06:35:42.560ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
47712025-04-30T06:35:42.561ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
47722025-04-30T06:35:42.564ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9353ce3e-c41a-460c-b9e3-1a6ea5e9f37d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47732025-04-30T06:35:42.565ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
47742025-04-30T06:35:42.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1563c65-4532-4ae3-bfdb-7495330127df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47752025-04-30T06:35:42.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
47762025-04-30T06:35:42.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47772025-04-30T06:35:42.567ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47782025-04-30T06:35:42.568ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
47792025-04-30T06:35:42.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
47802025-04-30T06:35:42.570ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5facceb1-b302-4b97-a4eb-abadf879e943", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47812025-04-30T06:35:42.571ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
47822025-04-30T06:35:42.574ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
47832025-04-30T06:35:42.574ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
47842025-04-30T06:35:42.575ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76768cc6-520b-454d-8c6f-ffbc4cd22b6c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47852025-04-30T06:35:42.575ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
47862025-04-30T06:35:42.577ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "59dce6ee-b94f-4aad-b596-7edc58b78201", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47872025-04-30T06:35:42.577ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
47882025-04-30T06:35:42.584ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
47892025-04-30T06:35:42.585ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
47902025-04-30T06:35:42.586ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c441072-9196-427d-b452-510372d03d0f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47912025-04-30T06:35:42.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
47922025-04-30T06:35:42.589ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5ea7f54-4038-4da8-8f36-e1d54759280e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47932025-04-30T06:35:42.590ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
47942025-04-30T06:35:42.598ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a899d90-2bb3-4cc6-8055-ccf94e0c1bcc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47952025-04-30T06:35:42.598ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
47962025-04-30T06:35:42.602ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4390800a-1e2b-40ad-8e40-d49b48936ace", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47972025-04-30T06:35:42.602ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
47982025-04-30T06:35:42.605ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b98cbcb-b333-4a35-877f-a74be9ff0fee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
47992025-04-30T06:35:42.606ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
48002025-04-30T06:35:42.610ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34ae1457-abfc-4b12-973b-8f95d7bbbae4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48012025-04-30T06:35:42.610ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
48022025-04-30T06:35:42.612ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48032025-04-30T06:35:42.612ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48042025-04-30T06:35:42.612ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48052025-04-30T06:35:42.612ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48062025-04-30T06:35:42.613ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48072025-04-30T06:35:42.615ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "588d3692-4d1e-40c2-ba90-22da368b1db1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48082025-04-30T06:35:42.615ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
48092025-04-30T06:35:42.616ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48102025-04-30T06:35:42.622ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48112025-04-30T06:35:42.622ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48122025-04-30T06:35:42.622ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48132025-04-30T06:35:42.623ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48142025-04-30T06:35:42.624ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d2a6d32-c8d6-4af5-85e9-f7de8c906811", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48152025-04-30T06:35:42.625ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
48162025-04-30T06:35:42.626ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "497cc089-f838-414a-92c6-c7006ab0ddad", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48172025-04-30T06:35:42.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
48182025-04-30T06:35:42.627ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48192025-04-30T06:35:42.627ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48202025-04-30T06:35:42.628ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48212025-04-30T06:35:42.630ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf370bc6-34bc-41ac-97dc-2bb41a535013", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48222025-04-30T06:35:42.630ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
48232025-04-30T06:35:42.632ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a203cde6-769b-4279-ac8e-34a0bdcad6b2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48242025-04-30T06:35:42.632ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
48252025-04-30T06:35:42.633ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d945c0-48a8-46ae-91a6-9aabe8dd57f8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48262025-04-30T06:35:42.634ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
48272025-04-30T06:35:42.636ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a187157-1b80-4569-9536-fd8ae3b4cddc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48282025-04-30T06:35:42.636ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
48292025-04-30T06:35:42.639ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54ccd5a4-a2f5-45b1-9af8-1de74e804bd7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48302025-04-30T06:35:42.639ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
48312025-04-30T06:35:42.642ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "794fb2ff-21af-430a-83bf-4801ebcec7ec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48322025-04-30T06:35:42.643ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
48332025-04-30T06:35:42.648ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fca5978-d8d1-47b1-b959-05dbdb289180", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48342025-04-30T06:35:42.648ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
48352025-04-30T06:35:42.649ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "99c7f54e-3e94-4fd6-8229-9b80ff6f8464", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48362025-04-30T06:35:42.650ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
48372025-04-30T06:35:42.653ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e4e3b98-07bd-438c-ab24-c1378500320c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48382025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
48392025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3138187-21f0-4bfa-846e-9f3372032e18", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48402025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
48412025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc75018c-ca33-4bbb-957b-7c3e6c7ecdcb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48422025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
48432025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8db7bcf-1b11-4cb0-83ca-6ba5f57b7e01", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48442025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
48452025-04-30T06:35:42.674ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
48462025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "03c225b5-5afa-4d65-89b0-514e00a00e9b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48472025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
48482025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a48e8e15-5bc3-4809-9da2-f658387d8aab", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48492025-04-30T06:35:42.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
48502025-04-30T06:35:42.683ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
48512025-04-30T06:35:42.684ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7305d964-e99a-4dd4-812e-16ce353d4270", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48522025-04-30T06:35:42.684ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
48532025-04-30T06:35:42.687ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cdd6065-a9c4-4ff1-bcca-f40ffe870fa7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48542025-04-30T06:35:42.687ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
48552025-04-30T06:35:42.692ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb0adc5b-8d21-4b47-ab56-b6b5503b3a0b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48562025-04-30T06:35:42.693ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
48572025-04-30T06:35:42.696ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d4c3f6e-0152-4aec-84ec-b25be060148f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48582025-04-30T06:35:42.696ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
48592025-04-30T06:35:42.697ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfb35f38-2323-48da-b175-288a2ada3944", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48602025-04-30T06:35:42.697ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
48612025-04-30T06:35:42.702ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65330787-bffa-490b-8800-a95c2c1e5582", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48622025-04-30T06:35:42.702ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
48632025-04-30T06:35:42.703ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6d61611-cf2e-46fc-84f5-73dbdd3a038a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48642025-04-30T06:35:42.703ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
48652025-04-30T06:35:42.704ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d893ec7c-d189-4618-bd3d-2093b76da735", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48662025-04-30T06:35:42.704ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
48672025-04-30T06:35:42.708ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0058ebd3-267e-4cca-b658-3511623fa621", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48682025-04-30T06:35:42.709ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
48692025-04-30T06:35:42.714ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd6741e7-d13d-45ac-b2f5-b4201c123c86", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
48702025-04-30T06:35:42.714ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48712025-04-30T06:35:42.714ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48722025-04-30T06:35:42.714ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48732025-04-30T06:35:42.714ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48742025-04-30T06:35:42.715ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48752025-04-30T06:35:42.715ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
48762025-04-30T06:35:42.715ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48772025-04-30T06:35:42.715ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48782025-04-30T06:35:42.715ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48792025-04-30T06:35:42.715ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48802025-04-30T06:35:42.716ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48812025-04-30T06:35:42.722ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48822025-04-30T06:35:42.722ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
48832025-04-30T06:35:42.722ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
48842025-04-30T06:35:42.722ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48852025-04-30T06:35:42.723ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48862025-04-30T06:35:42.723ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48872025-04-30T06:35:42.728ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48882025-04-30T06:35:42.740ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
48892025-04-30T06:35:42.740ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48902025-04-30T06:35:42.741ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48912025-04-30T06:35:42.742ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
48922025-04-30T06:35:42.744ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
48932025-04-30T06:35:42.749ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
48942025-04-30T06:35:42.760ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
48952025-04-30T06:35:42.760ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
48962025-04-30T06:35:42.761ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
48972025-04-30T06:35:42.773ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
48982025-04-30T06:35:42.773ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
48992025-04-30T06:35:42.773ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49002025-04-30T06:35:42.836ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
49012025-04-30T06:35:42.836ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
49022025-04-30T06:35:42.837ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:44597 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
49032025-04-30T06:35:42.838ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:44597 req_id = fde94a4b-4459-4e1f-9d6a-5d74d905871e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49042025-04-30T06:35:42.838ZINFOtest_metrics (dropshot): request completed latency_us = 687 local_addr = [::1]:56237 method = GET remote_addr = [::1]:44597 req_id = fde94a4b-4459-4e1f-9d6a-5d74d905871e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
49052025-04-30T06:35:42.840ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "fde94a4b-4459-4e1f-9d6a-5d74d905871e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49062025-04-30T06:35:42.843ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49072025-04-30T06:35:42.843ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
49082025-04-30T06:35:42.843ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
49092025-04-30T06:35:42.846ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49102025-04-30T06:35:42.846ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49112025-04-30T06:35:42.846ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49122025-04-30T06:35:42.847ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49132025-04-30T06:35:42.848ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49142025-04-30T06:35:42.853ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49152025-04-30T06:35:42.856ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49162025-04-30T06:35:42.857ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49172025-04-30T06:35:42.857ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49182025-04-30T06:35:42.857ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49192025-04-30T06:35:42.898ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
49202025-04-30T06:35:42.903ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
49212025-04-30T06:35:42.904ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
49222025-04-30T06:35:42.904ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 498.410631ms iteration = 10 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
49232025-04-30T06:35:42.904ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 11 reason = Signaled
49242025-04-30T06:35:42.904ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
49252025-04-30T06:35:42.904ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49262025-04-30T06:35:42.904ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49272025-04-30T06:35:42.904ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49282025-04-30T06:35:42.905ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49292025-04-30T06:35:42.908ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49302025-04-30T06:35:42.913ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
49312025-04-30T06:35:42.913ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
49322025-04-30T06:35:42.913ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
49332025-04-30T06:35:42.913ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
49342025-04-30T06:35:42.918ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
49352025-04-30T06:35:42.918ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
49362025-04-30T06:35:42.919ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 50 de 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 P............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56216 req_id = d3f1ee82-7b6c-4d88-81a5-3873c596d186
49372025-04-30T06:35:42.919ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20702,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56216 req_id = d3f1ee82-7b6c-4d88-81a5-3873c596d186
49382025-04-30T06:35:42.920ZDEBGtest_metrics (store): query key key = _mgs._tcp
49392025-04-30T06:35:42.920ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49402025-04-30T06:35:42.920ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56216 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d3f1ee82-7b6c-4d88-81a5-3873c596d186
49412025-04-30T06:35:42.921ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3807, tv_nsec: 720161336 } })
49422025-04-30T06:35:42.921ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
49432025-04-30T06:35:42.922ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: af 09 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60881 req_id = 84f37729-e0d3-4e65-bc79-7f4fde576cad
49442025-04-30T06:35:42.922ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44809,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60881 req_id = 84f37729-e0d3-4e65-bc79-7f4fde576cad
49452025-04-30T06:35:42.922ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
49462025-04-30T06:35:42.922ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60881 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 84f37729-e0d3-4e65-bc79-7f4fde576cad
49472025-04-30T06:35:42.979ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
49482025-04-30T06:35:42.980ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
49492025-04-30T06:35:42.980ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
49502025-04-30T06:35:42.980ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49512025-04-30T06:35:42.980ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
49522025-04-30T06:35:42.981ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49532025-04-30T06:35:42.981ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 06 03 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63767 req_id = c37b01bc-64f0-4377-923c-751955eac2d4
49542025-04-30T06:35:42.981ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1539,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63767 req_id = c37b01bc-64f0-4377-923c-751955eac2d4
49552025-04-30T06:35:42.981ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
49562025-04-30T06:35:42.982ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 1539, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63767 req_id = c37b01bc-64f0-4377-923c-751955eac2d4
49572025-04-30T06:35:42.983ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
49582025-04-30T06:35:42.983ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
49592025-04-30T06:35:42.983ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
49602025-04-30T06:35:42.984ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
49612025-04-30T06:35:42.990ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
49622025-04-30T06:35:42.990ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49632025-04-30T06:35:42.990ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
49642025-04-30T06:35:42.992ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49652025-04-30T06:35:42.994ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "300bdcf1-ba18-4f14-a779-2f90ac6b41ff", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49662025-04-30T06:35:42.995ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
49672025-04-30T06:35:42.995ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
49682025-04-30T06:35:42.995ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
49692025-04-30T06:35:42.997ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
49702025-04-30T06:35:43.001ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d12e9836-573d-4cad-9aaf-b1e0e34c2160", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49712025-04-30T06:35:43.001ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
49722025-04-30T06:35:43.008ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aef7054c-e9a4-4597-a977-06ff6f85477f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49732025-04-30T06:35:43.008ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
49742025-04-30T06:35:43.010ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27da1447-a0f0-4e16-85ad-7b3b40618c4b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49752025-04-30T06:35:43.010ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
49762025-04-30T06:35:43.012ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "17e3775e-b9cd-49a5-9a05-ce54781a0826", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49772025-04-30T06:35:43.013ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
49782025-04-30T06:35:43.015ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f815aa4c-2000-4fc3-bf09-8c4e537622a5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49792025-04-30T06:35:43.015ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
49802025-04-30T06:35:43.017ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7986120-ac5f-4ba7-a3ce-0eadf0d85769", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49812025-04-30T06:35:43.018ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
49822025-04-30T06:35:43.019ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "394f0833-61fa-41af-b08a-0b1b1d23341f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49832025-04-30T06:35:43.020ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
49842025-04-30T06:35:43.021ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "006ffc19-3c9a-4f33-a44b-d5f9fc5e8a70", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49852025-04-30T06:35:43.021ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
49862025-04-30T06:35:43.029ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "19349482-3740-4b68-ba1f-8c080d6a64bf", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49872025-04-30T06:35:43.029ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
49882025-04-30T06:35:43.034ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
49892025-04-30T06:35:43.035ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e6ee23a-e646-47b7-9a27-b9cb444c4081", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49902025-04-30T06:35:43.035ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
49912025-04-30T06:35:43.037ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9398ca5a-ecb1-4d40-920a-91363bcd119b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49922025-04-30T06:35:43.037ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
49932025-04-30T06:35:43.039ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1122bc22-494c-46ad-9847-2fa2e1ae6a7b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49942025-04-30T06:35:43.039ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
49952025-04-30T06:35:43.042ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4db205db-5990-436c-9ec7-180500b18bc9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49962025-04-30T06:35:43.043ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
49972025-04-30T06:35:43.046ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db081fb6-16a5-4f14-b7de-f051f7a42e31", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
49982025-04-30T06:35:43.046ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
49992025-04-30T06:35:43.050ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "67eb93c6-2000-48b7-9a3a-57c1c5023ab7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50002025-04-30T06:35:43.051ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
50012025-04-30T06:35:43.053ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4d87fb0-6c81-406d-b6fa-f8f9709718ae", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50022025-04-30T06:35:43.054ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
50032025-04-30T06:35:43.057ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdfbac0e-b55f-4977-b26b-51ad006bdeec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50042025-04-30T06:35:43.058ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
50052025-04-30T06:35:43.061ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "097b3c9b-4058-4e09-a6f3-56dce4c95067", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50062025-04-30T06:35:43.061ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
50072025-04-30T06:35:43.063ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "515ca31a-0b56-43e2-9610-e31dd5050d08", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50082025-04-30T06:35:43.063ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
50092025-04-30T06:35:43.066ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5c9d444-f0c1-4b16-986d-10527bd37f29", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50102025-04-30T06:35:43.066ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
50112025-04-30T06:35:43.070ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e599c5f1-d367-4fe1-95a0-e93538d414ca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50122025-04-30T06:35:43.070ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
50132025-04-30T06:35:43.072ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "abb76a7f-0ff6-47f5-8f6b-b0115d37860e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50142025-04-30T06:35:43.072ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
50152025-04-30T06:35:43.075ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7ea48d5-5cf5-4631-8ddc-b5b6336b22d8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50162025-04-30T06:35:43.075ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
50172025-04-30T06:35:43.079ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2130a54-e692-4ca4-b697-3fee9c27fd5e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50182025-04-30T06:35:43.079ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
50192025-04-30T06:35:43.083ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c835a46d-ec13-44f3-bc5b-500575217884", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50202025-04-30T06:35:43.083ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
50212025-04-30T06:35:43.086ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50222025-04-30T06:35:43.086ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50232025-04-30T06:35:43.086ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50242025-04-30T06:35:43.086ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50252025-04-30T06:35:43.089ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50262025-04-30T06:35:43.099ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b3b9a2b-2333-4dc7-87ec-57515bdc8453", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50272025-04-30T06:35:43.099ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
50282025-04-30T06:35:43.100ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50292025-04-30T06:35:43.107ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50302025-04-30T06:35:43.107ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50312025-04-30T06:35:43.107ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50322025-04-30T06:35:43.107ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50332025-04-30T06:35:43.109ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50342025-04-30T06:35:43.110ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50352025-04-30T06:35:43.110ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50362025-04-30T06:35:43.111ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7885deec-7773-4f40-ae39-2ff17b1746d0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50372025-04-30T06:35:43.111ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
50382025-04-30T06:35:43.115ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb8c18e5-e10f-480c-96e8-7fd4c961b719", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50392025-04-30T06:35:43.115ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
50402025-04-30T06:35:43.117ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0b49e3f-e3a6-46fd-bb52-d566f4413fba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50412025-04-30T06:35:43.117ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
50422025-04-30T06:35:43.118ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e1616e-2820-4d14-b110-f4cb86307c8d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50432025-04-30T06:35:43.118ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
50442025-04-30T06:35:43.119ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "21376d89-4b77-4c48-a77c-7790d421037e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50452025-04-30T06:35:43.119ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
50462025-04-30T06:35:43.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7275426a-df40-40c3-bb9a-77a64ce3bf0d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50472025-04-30T06:35:43.121ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
50482025-04-30T06:35:43.122ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70196037-4fb4-4eb7-a2c4-b6258ab8aa32", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50492025-04-30T06:35:43.123ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
50502025-04-30T06:35:43.124ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6bf6329-a012-4be5-b84e-e08a022c2264", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50512025-04-30T06:35:43.124ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
50522025-04-30T06:35:43.128ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d2e18a4-0bd3-4608-8c26-3d585b3d0a19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50532025-04-30T06:35:43.129ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
50542025-04-30T06:35:43.131ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "164ba98b-109e-44f7-b753-227dc13887ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50552025-04-30T06:35:43.131ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
50562025-04-30T06:35:43.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d8f252d-ca5f-449e-83bf-365628ecdbc8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50572025-04-30T06:35:43.135ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
50582025-04-30T06:35:43.140ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9772d30f-7c39-4f1b-ac37-4e64b347af6b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50592025-04-30T06:35:43.140ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
50602025-04-30T06:35:43.142ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
50612025-04-30T06:35:43.147ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50724b40-c7ac-44de-8836-dafc15eff326", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50622025-04-30T06:35:43.147ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
50632025-04-30T06:35:43.149ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a27eb89d-c0a5-41ae-9001-f0a24131fc9c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50642025-04-30T06:35:43.149ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
50652025-04-30T06:35:43.150ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c038012-da89-496c-937f-fc1b0f63d577", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50662025-04-30T06:35:43.150ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
50672025-04-30T06:35:43.165ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f315de64-5f1a-45e3-af7c-f7c6dfeb0642", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50682025-04-30T06:35:43.165ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
50692025-04-30T06:35:43.167ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "602eef06-543f-4bb1-83ad-85f7cf3f02cc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50702025-04-30T06:35:43.167ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
50712025-04-30T06:35:43.169ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ab7a791-7fb8-479b-bd36-f04c08d9bc48", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
50722025-04-30T06:35:43.169ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50732025-04-30T06:35:43.169ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50742025-04-30T06:35:43.169ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50752025-04-30T06:35:43.169ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50762025-04-30T06:35:43.170ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50772025-04-30T06:35:43.179ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50782025-04-30T06:35:43.189ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50792025-04-30T06:35:43.190ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
50802025-04-30T06:35:43.190ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50812025-04-30T06:35:43.190ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50822025-04-30T06:35:43.190ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50832025-04-30T06:35:43.191ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50842025-04-30T06:35:43.193ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
50852025-04-30T06:35:43.193ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
50862025-04-30T06:35:43.193ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50872025-04-30T06:35:43.195ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50882025-04-30T06:35:43.200ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50892025-04-30T06:35:43.204ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50902025-04-30T06:35:43.209ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
50912025-04-30T06:35:43.209ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
50922025-04-30T06:35:43.209ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
50932025-04-30T06:35:43.209ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
50942025-04-30T06:35:43.211ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
50952025-04-30T06:35:43.219ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
50962025-04-30T06:35:43.219ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
50972025-04-30T06:35:43.220ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
50982025-04-30T06:35:43.220ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
50992025-04-30T06:35:43.277ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
51002025-04-30T06:35:43.277ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
51012025-04-30T06:35:43.278ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:33867 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
51022025-04-30T06:35:43.279ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:33867 req_id = 366f8603-0c89-43b8-a2b0-0492131a4f8f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51032025-04-30T06:35:43.279ZINFOtest_metrics (dropshot): request completed latency_us = 754 local_addr = [::1]:56237 method = GET remote_addr = [::1]:33867 req_id = 366f8603-0c89-43b8-a2b0-0492131a4f8f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
51042025-04-30T06:35:43.280ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "366f8603-0c89-43b8-a2b0-0492131a4f8f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51052025-04-30T06:35:43.281ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51062025-04-30T06:35:43.281ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
51072025-04-30T06:35:43.281ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
51082025-04-30T06:35:43.284ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51092025-04-30T06:35:43.284ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51102025-04-30T06:35:43.284ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51112025-04-30T06:35:43.284ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51122025-04-30T06:35:43.289ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51132025-04-30T06:35:43.289ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51142025-04-30T06:35:43.289ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51152025-04-30T06:35:43.290ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51162025-04-30T06:35:43.301ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51172025-04-30T06:35:43.312ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51182025-04-30T06:35:43.312ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51192025-04-30T06:35:43.312ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51202025-04-30T06:35:43.313ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51212025-04-30T06:35:43.314ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
51222025-04-30T06:35:43.314ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51232025-04-30T06:35:43.315ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 47d2c08e-6a35-469a-8bfc-f839c5fee69b uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51242025-04-30T06:35:43.316ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 47d2c08e-6a35-469a-8bfc-f839c5fee69b resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51252025-04-30T06:35:43.316ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 47d2c08e-6a35-469a-8bfc-f839c5fee69b roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51262025-04-30T06:35:43.317ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 47d2c08e-6a35-469a-8bfc-f839c5fee69b resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51272025-04-30T06:35:43.321ZINFOtest_metrics (dropshot_internal): request completed latency_us = 5184 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 47d2c08e-6a35-469a-8bfc-f839c5fee69b response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
51282025-04-30T06:35:43.321ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "47d2c08e-6a35-469a-8bfc-f839c5fee69b", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51292025-04-30T06:35:43.321ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51302025-04-30T06:35:43.327ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83bf1671-b638-40fa-bed0-fc1dec7685b5 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51312025-04-30T06:35:43.327ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83bf1671-b638-40fa-bed0-fc1dec7685b5 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51322025-04-30T06:35:43.328ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83bf1671-b638-40fa-bed0-fc1dec7685b5 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51332025-04-30T06:35:43.328ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83bf1671-b638-40fa-bed0-fc1dec7685b5 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51342025-04-30T06:35:43.332ZINFOtest_metrics (dropshot_internal): request completed latency_us = 5473 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83bf1671-b638-40fa-bed0-fc1dec7685b5 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
51352025-04-30T06:35:43.333ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "83bf1671-b638-40fa-bed0-fc1dec7685b5", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51362025-04-30T06:35:43.334ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
51372025-04-30T06:35:43.334ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
51382025-04-30T06:35:43.334ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
51392025-04-30T06:35:43.361ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
51402025-04-30T06:35:43.361ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
51412025-04-30T06:35:43.361ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 457.227286ms iteration = 11 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
51422025-04-30T06:35:43.361ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 12 reason = Signaled
51432025-04-30T06:35:43.361ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
51442025-04-30T06:35:43.361ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51452025-04-30T06:35:43.361ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51462025-04-30T06:35:43.361ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51472025-04-30T06:35:43.362ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51482025-04-30T06:35:43.376ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51492025-04-30T06:35:43.381ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
51502025-04-30T06:35:43.381ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
51512025-04-30T06:35:43.381ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
51522025-04-30T06:35:43.381ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
51532025-04-30T06:35:43.387ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
51542025-04-30T06:35:43.392ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
51552025-04-30T06:35:43.392ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
51562025-04-30T06:35:43.393ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 60 32 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 `2..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61037 req_id = deefd08e-3c20-48ff-8aa7-fe1a41128434
51572025-04-30T06:35:43.393ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24626,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61037 req_id = deefd08e-3c20-48ff-8aa7-fe1a41128434
51582025-04-30T06:35:43.393ZDEBGtest_metrics (store): query key key = _mgs._tcp
51592025-04-30T06:35:43.394ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51602025-04-30T06:35:43.394ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61037 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = deefd08e-3c20-48ff-8aa7-fe1a41128434
51612025-04-30T06:35:43.394ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3808, tv_nsec: 193693558 } })
51622025-04-30T06:35:43.394ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
51632025-04-30T06:35:43.395ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: df 76 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .v..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58403 req_id = 5a8d684e-84df-4a9b-95a3-6526ae49e871
51642025-04-30T06:35:43.395ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57206,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58403 req_id = 5a8d684e-84df-4a9b-95a3-6526ae49e871
51652025-04-30T06:35:43.395ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
51662025-04-30T06:35:43.395ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58403 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5a8d684e-84df-4a9b-95a3-6526ae49e871
51672025-04-30T06:35:43.451ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
51682025-04-30T06:35:43.451ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
51692025-04-30T06:35:43.451ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
51702025-04-30T06:35:43.452ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51712025-04-30T06:35:43.452ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51722025-04-30T06:35:43.452ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51732025-04-30T06:35:43.453ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2f 56 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c /V..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54262 req_id = 05738ddd-ece4-42c0-8f63-31b2451f82cd
51742025-04-30T06:35:43.453ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12118,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54262 req_id = 05738ddd-ece4-42c0-8f63-31b2451f82cd
51752025-04-30T06:35:43.453ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
51762025-04-30T06:35:43.454ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 12118, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54262 req_id = 05738ddd-ece4-42c0-8f63-31b2451f82cd
51772025-04-30T06:35:43.454ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
51782025-04-30T06:35:43.454ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
51792025-04-30T06:35:43.454ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
51802025-04-30T06:35:43.455ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
51812025-04-30T06:35:43.461ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
51822025-04-30T06:35:43.461ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51832025-04-30T06:35:43.461ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51842025-04-30T06:35:43.462ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51852025-04-30T06:35:43.464ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9374aa02-a691-48d3-86f2-baaef21694b7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51862025-04-30T06:35:43.464ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
51872025-04-30T06:35:43.466ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
51882025-04-30T06:35:43.466ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
51892025-04-30T06:35:43.467ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
51902025-04-30T06:35:43.467ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eb31a6a-61f7-4fda-9ff8-b520f2e1b8fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51912025-04-30T06:35:43.467ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
51922025-04-30T06:35:43.470ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1000a4c5-814d-4b3a-9d4a-1a277ed6db7b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51932025-04-30T06:35:43.470ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
51942025-04-30T06:35:43.474ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77b4d823-87ef-46f1-9aba-c31682fe14dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51952025-04-30T06:35:43.474ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
51962025-04-30T06:35:43.488ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f9720b1-c5d1-4403-ad94-bacc32d667fa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51972025-04-30T06:35:43.493ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
51982025-04-30T06:35:43.499ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1aff501b-d44c-4aae-959b-957653ac3a43", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
51992025-04-30T06:35:43.499ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
52002025-04-30T06:35:43.505ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3618c1ce-b38f-4d2c-b667-ecf78b5bb834", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:42 GMT"} })
52012025-04-30T06:35:43.505ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
52022025-04-30T06:35:43.511ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "458a01e5-15e0-448c-a338-3035f09c443e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52032025-04-30T06:35:43.511ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
52042025-04-30T06:35:43.512ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
52052025-04-30T06:35:43.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52062025-04-30T06:35:43.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52072025-04-30T06:35:43.514ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93153ba8-c029-4265-bb31-3044ea1947f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52082025-04-30T06:35:43.514ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
52092025-04-30T06:35:43.516ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52102025-04-30T06:35:43.517ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52112025-04-30T06:35:43.517ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52122025-04-30T06:35:43.517ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
52132025-04-30T06:35:43.518ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ee35d0c-c8da-4a08-a37e-42d9cf92d173", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52142025-04-30T06:35:43.518ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
52152025-04-30T06:35:43.519ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6251246f-03d5-4d33-9eea-9febaaab0016", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52162025-04-30T06:35:43.519ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
52172025-04-30T06:35:43.520ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ca0439-e0e0-43b5-a07e-8b79773531a1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52182025-04-30T06:35:43.523ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
52192025-04-30T06:35:43.524ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a6e7229-7ae1-4d26-8b89-37d0ad587ebb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52202025-04-30T06:35:43.525ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
52212025-04-30T06:35:43.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b165ceec-7b30-41c2-b330-3acce240617d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52222025-04-30T06:35:43.527ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
52232025-04-30T06:35:43.552ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acd22b14-ef83-4edd-8289-d55211db6493", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52242025-04-30T06:35:43.552ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
52252025-04-30T06:35:43.554ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa580306-31f2-4322-ba12-6475107b5683", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52262025-04-30T06:35:43.554ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
52272025-04-30T06:35:43.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3f1822-80de-47e0-8658-0b5dc4addd71", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52282025-04-30T06:35:43.556ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
52292025-04-30T06:35:43.562ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55c9f659-813a-46cb-a3e9-48e4eb14e2e4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52302025-04-30T06:35:43.562ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
52312025-04-30T06:35:43.563ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
52322025-04-30T06:35:43.563ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
52332025-04-30T06:35:43.563ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52342025-04-30T06:35:43.563ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52352025-04-30T06:35:43.564ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52362025-04-30T06:35:43.566ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
52372025-04-30T06:35:43.572ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
52382025-04-30T06:35:43.572ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52392025-04-30T06:35:43.572ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52402025-04-30T06:35:43.572ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52412025-04-30T06:35:43.580ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52422025-04-30T06:35:43.580ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52432025-04-30T06:35:43.581ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
52442025-04-30T06:35:43.581ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
52452025-04-30T06:35:43.583ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
52462025-04-30T06:35:43.584ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
52472025-04-30T06:35:43.584ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
52482025-04-30T06:35:43.585ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18d8b993-357e-492d-ad5f-bfbe6bddcf18", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52492025-04-30T06:35:43.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
52502025-04-30T06:35:43.587ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
52512025-04-30T06:35:43.587ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
52522025-04-30T06:35:43.587ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92945b4f-71fc-4d80-a314-79fa9fd64573", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52532025-04-30T06:35:43.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
52542025-04-30T06:35:43.589ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "56bdb3de-1ab4-430e-a67b-6e18f70b8ed8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52552025-04-30T06:35:43.589ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
52562025-04-30T06:35:43.591ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
52572025-04-30T06:35:43.591ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
52582025-04-30T06:35:43.592ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dfe126e-2812-4641-88d7-598d7eed32fe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52592025-04-30T06:35:43.592ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
52602025-04-30T06:35:43.593ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "495e5c1c-183c-4c03-a131-726a4bf4832e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52612025-04-30T06:35:43.593ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
52622025-04-30T06:35:43.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a5601cb-c7f0-44c1-9ca4-2b6b79cdf1b3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52632025-04-30T06:35:43.597ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
52642025-04-30T06:35:43.599ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f7504b-2e0d-412c-b47c-7b9b63ea04f6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52652025-04-30T06:35:43.599ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
52662025-04-30T06:35:43.601ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "698e9429-d5e5-42c0-8804-52aa09efc7cf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52672025-04-30T06:35:43.601ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
52682025-04-30T06:35:43.608ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b71d50e-049a-4f30-853e-74ab977ddd57", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52692025-04-30T06:35:43.608ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
52702025-04-30T06:35:43.611ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "10041589-0698-4e6e-9f97-b455d5a8f19d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52712025-04-30T06:35:43.611ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
52722025-04-30T06:35:43.613ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1a2ccdd-c668-4c90-b3b3-bae1e4fa5b96", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52732025-04-30T06:35:43.613ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
52742025-04-30T06:35:43.616ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
52752025-04-30T06:35:43.618ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5d5b7ae-d2d3-4ea0-af39-31b7a15f61f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52762025-04-30T06:35:43.619ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
52772025-04-30T06:35:43.620ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "51a92c87-ef68-49a0-b786-5648faadf862", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52782025-04-30T06:35:43.621ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
52792025-04-30T06:35:43.622ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26868c1a-8155-47e7-9239-0d765cec1911", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52802025-04-30T06:35:43.622ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
52812025-04-30T06:35:43.623ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c2787e5-4aaa-4799-a3be-793b4d7b8db7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52822025-04-30T06:35:43.623ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
52832025-04-30T06:35:43.624ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ce7d52d-9adc-46fd-920b-e0dfae51fd07", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52842025-04-30T06:35:43.624ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
52852025-04-30T06:35:43.625ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b59bae22-ca5c-4c71-8472-107dd22f374b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52862025-04-30T06:35:43.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
52872025-04-30T06:35:43.630ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "da7c96f4-e82a-43c8-8fb2-9dd0fab379cc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52882025-04-30T06:35:43.630ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
52892025-04-30T06:35:43.632ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e538cfaa-81ff-46f8-8564-5e9f7c88aeda", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52902025-04-30T06:35:43.632ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
52912025-04-30T06:35:43.635ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd34b43f-f26b-47d0-b14b-891516438dc2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52922025-04-30T06:35:43.635ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
52932025-04-30T06:35:43.638ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2744f3be-a4ca-4a15-845b-4449bead6e69", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52942025-04-30T06:35:43.638ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
52952025-04-30T06:35:43.640ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b17b71f-acfe-419b-9c7c-58d6d72193c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52962025-04-30T06:35:43.640ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
52972025-04-30T06:35:43.642ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d7dd450-54ea-49af-9276-b1b8b39bfded", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
52982025-04-30T06:35:43.642ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
52992025-04-30T06:35:43.643ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22d11eab-2b48-49c7-ae00-8e6506298770", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
53002025-04-30T06:35:43.643ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
53012025-04-30T06:35:43.644ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "987ebcd3-6333-462a-9769-b112618446f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
53022025-04-30T06:35:43.644ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
53032025-04-30T06:35:43.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aef25852-50d5-4aac-b844-0bbb7b1d21d4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
53042025-04-30T06:35:43.646ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
53052025-04-30T06:35:43.647ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f291be40-c7e2-41f3-88c1-9786f0b04d4a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
53062025-04-30T06:35:43.647ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53072025-04-30T06:35:43.647ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53082025-04-30T06:35:43.647ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53092025-04-30T06:35:43.647ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53102025-04-30T06:35:43.648ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53112025-04-30T06:35:43.653ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53122025-04-30T06:35:43.657ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53132025-04-30T06:35:43.658ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
53142025-04-30T06:35:43.658ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53152025-04-30T06:35:43.658ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53162025-04-30T06:35:43.658ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53172025-04-30T06:35:43.659ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53182025-04-30T06:35:43.661ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53192025-04-30T06:35:43.666ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
53202025-04-30T06:35:43.666ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53212025-04-30T06:35:43.666ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53222025-04-30T06:35:43.666ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53232025-04-30T06:35:43.667ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53242025-04-30T06:35:43.667ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53252025-04-30T06:35:43.667ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53262025-04-30T06:35:43.667ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53272025-04-30T06:35:43.668ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53282025-04-30T06:35:43.672ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53292025-04-30T06:35:43.678ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53302025-04-30T06:35:43.678ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53312025-04-30T06:35:43.678ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53322025-04-30T06:35:43.679ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53332025-04-30T06:35:43.746ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
53342025-04-30T06:35:43.746ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
53352025-04-30T06:35:43.747ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:37027 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
53362025-04-30T06:35:43.748ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
53372025-04-30T06:35:43.748ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:37027 req_id = a0f3b0cb-cc1c-4b95-8b68-06886196068f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53382025-04-30T06:35:43.749ZINFOtest_metrics (dropshot): request completed latency_us = 581 local_addr = [::1]:56237 method = GET remote_addr = [::1]:37027 req_id = a0f3b0cb-cc1c-4b95-8b68-06886196068f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
53392025-04-30T06:35:43.749ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53402025-04-30T06:35:43.749ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53412025-04-30T06:35:43.750ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53422025-04-30T06:35:43.751ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0f3b0cb-cc1c-4b95-8b68-06886196068f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
53432025-04-30T06:35:43.753ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53442025-04-30T06:35:43.753ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
53452025-04-30T06:35:43.753ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
53462025-04-30T06:35:43.755ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53472025-04-30T06:35:43.755ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53482025-04-30T06:35:43.755ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53492025-04-30T06:35:43.755ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53502025-04-30T06:35:43.756ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53512025-04-30T06:35:43.764ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53522025-04-30T06:35:43.769ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53532025-04-30T06:35:43.770ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53542025-04-30T06:35:43.770ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53552025-04-30T06:35:43.771ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53562025-04-30T06:35:43.794ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
53572025-04-30T06:35:43.797ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
53582025-04-30T06:35:43.798ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
53592025-04-30T06:35:43.798ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 436.765376ms iteration = 12 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
53602025-04-30T06:35:43.798ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 13 reason = Signaled
53612025-04-30T06:35:43.798ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
53622025-04-30T06:35:43.798ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53632025-04-30T06:35:43.798ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53642025-04-30T06:35:43.798ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53652025-04-30T06:35:43.799ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53662025-04-30T06:35:43.801ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53672025-04-30T06:35:43.805ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
53682025-04-30T06:35:43.805ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
53692025-04-30T06:35:43.805ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
53702025-04-30T06:35:43.806ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
53712025-04-30T06:35:43.810ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
53722025-04-30T06:35:43.810ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
53732025-04-30T06:35:43.811ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e2 ea 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52285 req_id = e9ba3fb0-a267-4bac-a4a7-2aed0bed1783
53742025-04-30T06:35:43.812ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58090,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52285 req_id = e9ba3fb0-a267-4bac-a4a7-2aed0bed1783
53752025-04-30T06:35:43.812ZDEBGtest_metrics (store): query key key = _mgs._tcp
53762025-04-30T06:35:43.812ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53772025-04-30T06:35:43.812ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52285 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e9ba3fb0-a267-4bac-a4a7-2aed0bed1783
53782025-04-30T06:35:43.812ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3808, tv_nsec: 611693626 } })
53792025-04-30T06:35:43.812ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
53802025-04-30T06:35:43.813ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 6e 55 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e nU..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64584 req_id = 83e15ff0-1e90-4565-9eec-6f659d12ae29
53812025-04-30T06:35:43.813ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28245,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64584 req_id = 83e15ff0-1e90-4565-9eec-6f659d12ae29
53822025-04-30T06:35:43.813ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
53832025-04-30T06:35:43.813ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64584 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 83e15ff0-1e90-4565-9eec-6f659d12ae29
53842025-04-30T06:35:43.863ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
53852025-04-30T06:35:43.863ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
53862025-04-30T06:35:43.863ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
53872025-04-30T06:35:43.863ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
53882025-04-30T06:35:43.863ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
53892025-04-30T06:35:43.864ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
53902025-04-30T06:35:43.864ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 7f d3 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57429 req_id = 2827aae9-dfe4-49ad-87e2-c78db709dba2
53912025-04-30T06:35:43.864ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32723,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57429 req_id = 2827aae9-dfe4-49ad-87e2-c78db709dba2
53922025-04-30T06:35:43.865ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
53932025-04-30T06:35:43.865ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 32723, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57429 req_id = 2827aae9-dfe4-49ad-87e2-c78db709dba2
53942025-04-30T06:35:43.866ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
53952025-04-30T06:35:43.866ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
53962025-04-30T06:35:43.866ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
53972025-04-30T06:35:43.867ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
53982025-04-30T06:35:43.873ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
53992025-04-30T06:35:43.873ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54002025-04-30T06:35:43.873ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54012025-04-30T06:35:43.875ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54022025-04-30T06:35:43.878ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0801bc35-241f-4cf1-acda-5a592513094d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54032025-04-30T06:35:43.878ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
54042025-04-30T06:35:43.879ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54052025-04-30T06:35:43.879ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54062025-04-30T06:35:43.880ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54072025-04-30T06:35:43.886ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba398e99-80c9-4e3b-82aa-965e37c6727d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54082025-04-30T06:35:43.886ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
54092025-04-30T06:35:43.891ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fd028b3-0fbe-49d5-aa43-c8e55b2ce4e9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54102025-04-30T06:35:43.891ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
54112025-04-30T06:35:43.894ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e25f6fde-f642-4fc1-9cf5-05d0297e3e76", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54122025-04-30T06:35:43.894ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
54132025-04-30T06:35:43.896ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29bd3b00-c3de-4951-b30a-276246a4db25", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54142025-04-30T06:35:43.896ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
54152025-04-30T06:35:43.898ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a50142dc-11b1-4c52-8c62-4096376f06a4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54162025-04-30T06:35:43.899ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
54172025-04-30T06:35:43.901ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89a413a9-fc14-448a-b9eb-2df2b6bc389b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54182025-04-30T06:35:43.901ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
54192025-04-30T06:35:43.905ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6819fb9-e2bd-482f-8067-a30fab60c06e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54202025-04-30T06:35:43.905ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
54212025-04-30T06:35:43.906ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca4b6200-99ca-4465-8eff-2f27ebdb3fa5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54222025-04-30T06:35:43.907ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
54232025-04-30T06:35:43.909ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee253e3c-48b5-494c-b90b-6a106bee5b78", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54242025-04-30T06:35:43.909ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
54252025-04-30T06:35:43.910ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "467552f0-b936-4e48-b22f-071099efec90", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54262025-04-30T06:35:43.911ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
54272025-04-30T06:35:43.913ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a09cccb-05ba-4ab1-9ca9-9afbb30cdbc1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54282025-04-30T06:35:43.913ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
54292025-04-30T06:35:43.916ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "60584d83-e0ec-4ee1-b6b4-4afffa60b73d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54302025-04-30T06:35:43.916ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
54312025-04-30T06:35:43.924ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dafc7c86-c04d-407f-8682-37eb9d3b7a0d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54322025-04-30T06:35:43.925ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
54332025-04-30T06:35:43.925ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
54342025-04-30T06:35:43.929ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eaa3cea9-9157-45a6-b914-9d2f7fc9a68b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54352025-04-30T06:35:43.929ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
54362025-04-30T06:35:43.932ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64022d35-9c36-4982-9359-5090e74429a6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54372025-04-30T06:35:43.932ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
54382025-04-30T06:35:43.935ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0577ee17-4192-4478-8fab-f204ade5135f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54392025-04-30T06:35:43.935ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
54402025-04-30T06:35:43.937ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e233dc65-cab2-4ec5-acaf-7958603df835", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54412025-04-30T06:35:43.938ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
54422025-04-30T06:35:43.940ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f146813c-b74f-4392-9156-69fce6f5cb2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54432025-04-30T06:35:43.940ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
54442025-04-30T06:35:43.941ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a0d72634-0159-4d26-84b3-9471d558f106", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54452025-04-30T06:35:43.941ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
54462025-04-30T06:35:43.943ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d685878-7891-4fd4-b0b4-a25ff8040efc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54472025-04-30T06:35:43.943ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
54482025-04-30T06:35:43.944ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "037af061-cb4f-4012-8a33-1e0146674dd4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54492025-04-30T06:35:43.944ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
54502025-04-30T06:35:43.945ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "816b3d51-fdaa-40ec-968c-03be6373d937", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54512025-04-30T06:35:43.945ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
54522025-04-30T06:35:43.947ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f8d0595-7b07-40c9-90de-8fb4c70351c2", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54532025-04-30T06:35:43.947ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
54542025-04-30T06:35:43.949ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcbe4853-651f-410f-80b5-494ae10ab5d7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54552025-04-30T06:35:43.950ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
54562025-04-30T06:35:43.952ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "62075432-dd1b-49ea-a65b-f501def417a1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54572025-04-30T06:35:43.952ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
54582025-04-30T06:35:43.954ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9372313-df44-4242-9972-53cb806b39df", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54592025-04-30T06:35:43.955ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
54602025-04-30T06:35:43.956ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b905b6ef-b0a1-4920-bc45-2758557c3a8d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54612025-04-30T06:35:43.957ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
54622025-04-30T06:35:43.959ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c507057-6daa-4c02-8570-9c7e1cb87944", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54632025-04-30T06:35:43.959ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
54642025-04-30T06:35:43.961ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "175b974f-a7c8-499d-be5f-d886bd74fa63", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54652025-04-30T06:35:43.962ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
54662025-04-30T06:35:43.963ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "169c554a-d553-4692-9fcd-ed143a44e16c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54672025-04-30T06:35:43.963ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
54682025-04-30T06:35:43.964ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a006b040-bb7c-46a0-b64f-fd8256f04ef9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54692025-04-30T06:35:43.965ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
54702025-04-30T06:35:43.966ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4118aa1c-5603-4def-9e6c-a17b6f2b344e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54712025-04-30T06:35:43.966ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
54722025-04-30T06:35:43.967ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26d19192-71d6-45b7-84a5-730b27201bd7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54732025-04-30T06:35:43.967ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
54742025-04-30T06:35:43.968ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a45f55ae-5031-4de4-acc8-d7bec9cf0510", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54752025-04-30T06:35:43.968ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
54762025-04-30T06:35:43.973ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06099b16-b85b-4ae2-be37-54a66818ada7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54772025-04-30T06:35:43.973ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
54782025-04-30T06:35:43.975ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7fa4d09f-e187-4e1c-8694-1b9ad2dd14ae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54792025-04-30T06:35:43.975ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
54802025-04-30T06:35:43.976ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
54812025-04-30T06:35:43.976ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
54822025-04-30T06:35:43.976ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54832025-04-30T06:35:43.976ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54842025-04-30T06:35:43.977ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54852025-04-30T06:35:43.979ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "393747e9-1f31-4fb1-ad1f-b7d9471ce258", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54862025-04-30T06:35:43.980ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
54872025-04-30T06:35:43.981ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
54882025-04-30T06:35:43.986ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
54892025-04-30T06:35:43.987ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54902025-04-30T06:35:43.987ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54912025-04-30T06:35:43.987ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54922025-04-30T06:35:43.989ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b434d4b1-499d-4ed1-a251-edba9a47d2e1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54932025-04-30T06:35:43.989ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
54942025-04-30T06:35:43.989ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
54952025-04-30T06:35:43.990ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
54962025-04-30T06:35:43.990ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
54972025-04-30T06:35:43.994ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7653e4d2-aec0-490a-9b8a-7655a7a9c842", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
54982025-04-30T06:35:43.994ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
54992025-04-30T06:35:43.995ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4432f73b-9a84-494c-b3aa-e224a66a3c30", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55002025-04-30T06:35:43.995ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
55012025-04-30T06:35:43.997ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1e615b0-49e6-4643-a82d-bdb92cb7bc5f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55022025-04-30T06:35:43.997ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
55032025-04-30T06:35:43.999ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fbe29d36-5a06-4ae3-943f-c6d3981f1a46", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55042025-04-30T06:35:44.000ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
55052025-04-30T06:35:44.002ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8143379d-d6bd-4df3-9170-aca325415b01", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55062025-04-30T06:35:44.002ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
55072025-04-30T06:35:44.004ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21554c5-5cc4-40ba-8065-55a0ad89f838", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55082025-04-30T06:35:44.004ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55092025-04-30T06:35:44.004ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55102025-04-30T06:35:44.004ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55112025-04-30T06:35:44.004ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55122025-04-30T06:35:44.006ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55132025-04-30T06:35:44.008ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55142025-04-30T06:35:44.012ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55152025-04-30T06:35:44.013ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
55162025-04-30T06:35:44.013ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55172025-04-30T06:35:44.013ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55182025-04-30T06:35:44.013ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55192025-04-30T06:35:44.013ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55202025-04-30T06:35:44.016ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55212025-04-30T06:35:44.020ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
55222025-04-30T06:35:44.020ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55232025-04-30T06:35:44.020ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55242025-04-30T06:35:44.020ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55252025-04-30T06:35:44.096ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
55262025-04-30T06:35:44.096ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
55272025-04-30T06:35:44.098ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:48228 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
55282025-04-30T06:35:44.099ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:48228 req_id = b15435d7-3b27-4f30-a39f-38ecae947b4d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55292025-04-30T06:35:44.100ZINFOtest_metrics (dropshot): request completed latency_us = 499 local_addr = [::1]:56237 method = GET remote_addr = [::1]:48228 req_id = b15435d7-3b27-4f30-a39f-38ecae947b4d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
55302025-04-30T06:35:44.100ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b15435d7-3b27-4f30-a39f-38ecae947b4d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55312025-04-30T06:35:44.105ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55322025-04-30T06:35:44.105ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
55332025-04-30T06:35:44.105ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
55342025-04-30T06:35:44.108ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55352025-04-30T06:35:44.108ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55362025-04-30T06:35:44.108ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55372025-04-30T06:35:44.108ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55382025-04-30T06:35:44.111ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55392025-04-30T06:35:44.117ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55402025-04-30T06:35:44.125ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55412025-04-30T06:35:44.125ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55422025-04-30T06:35:44.126ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55432025-04-30T06:35:44.130ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55442025-04-30T06:35:44.136ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
55452025-04-30T06:35:44.180ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
55462025-04-30T06:35:44.181ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
55472025-04-30T06:35:44.181ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 383.448577ms iteration = 13 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
55482025-04-30T06:35:44.181ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 14 reason = Signaled
55492025-04-30T06:35:44.182ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
55502025-04-30T06:35:44.182ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55512025-04-30T06:35:44.182ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55522025-04-30T06:35:44.182ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55532025-04-30T06:35:44.190ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55542025-04-30T06:35:44.192ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
55552025-04-30T06:35:44.193ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
55562025-04-30T06:35:44.193ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55572025-04-30T06:35:44.193ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55582025-04-30T06:35:44.194ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55592025-04-30T06:35:44.195ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55602025-04-30T06:35:44.200ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
55612025-04-30T06:35:44.200ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
55622025-04-30T06:35:44.201ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
55632025-04-30T06:35:44.202ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
55642025-04-30T06:35:44.202ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
55652025-04-30T06:35:44.217ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
55662025-04-30T06:35:44.218ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55672025-04-30T06:35:44.218ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55682025-04-30T06:35:44.218ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55692025-04-30T06:35:44.221ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
55702025-04-30T06:35:44.221ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
55712025-04-30T06:35:44.222ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
55722025-04-30T06:35:44.223ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
55732025-04-30T06:35:44.223ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
55742025-04-30T06:35:44.224ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: fb 4f 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .O..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59277 req_id = 8221d1d2-1333-423e-b028-415ddde87878
55752025-04-30T06:35:44.224ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64335,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59277 req_id = 8221d1d2-1333-423e-b028-415ddde87878
55762025-04-30T06:35:44.224ZDEBGtest_metrics (store): query key key = _mgs._tcp
55772025-04-30T06:35:44.225ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55782025-04-30T06:35:44.225ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59277 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8221d1d2-1333-423e-b028-415ddde87878
55792025-04-30T06:35:44.226ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 25141213 } })
55802025-04-30T06:35:44.226ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
55812025-04-30T06:35:44.227ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: c2 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51339 req_id = b0150b74-eb26-423d-b312-e01c53c69926
55822025-04-30T06:35:44.227ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 49687,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51339 req_id = b0150b74-eb26-423d-b312-e01c53c69926
55832025-04-30T06:35:44.228ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
55842025-04-30T06:35:44.228ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51339 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b0150b74-eb26-423d-b312-e01c53c69926
55852025-04-30T06:35:44.304ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
55862025-04-30T06:35:44.311ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: f4 ae 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55548 req_id = b27ae179-aaeb-4a85-85a0-7bd3740ff462
55872025-04-30T06:35:44.311ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62638,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55548 req_id = b27ae179-aaeb-4a85-85a0-7bd3740ff462
55882025-04-30T06:35:44.312ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
55892025-04-30T06:35:44.312ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 62638, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55548 req_id = b27ae179-aaeb-4a85-85a0-7bd3740ff462
55902025-04-30T06:35:44.312ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
55912025-04-30T06:35:44.312ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
55922025-04-30T06:35:44.312ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
55932025-04-30T06:35:44.312ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3271a04e-8c95-4d1d-bcec-0a9c6bc93baa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55942025-04-30T06:35:44.316ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
55952025-04-30T06:35:44.317ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12802e0a-7f23-4887-9108-96b29a76cd61", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55962025-04-30T06:35:44.317ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
55972025-04-30T06:35:44.329ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43250b2b-f693-4952-adf7-c3e8683eff6c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
55982025-04-30T06:35:44.329ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
55992025-04-30T06:35:44.366ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
56002025-04-30T06:35:44.387ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5860c9b7-88ea-4bc2-b1bb-e0296abf6766", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56012025-04-30T06:35:44.387ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
56022025-04-30T06:35:44.390ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2769bde8-f460-46a1-bd03-b58460b217e0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56032025-04-30T06:35:44.390ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
56042025-04-30T06:35:44.393ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fe9734c-6b18-42fe-9817-ac8d28d44a8d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56052025-04-30T06:35:44.394ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
56062025-04-30T06:35:44.396ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ff616ce-598c-4670-8edd-6682a4f2645d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56072025-04-30T06:35:44.396ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
56082025-04-30T06:35:44.405ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56e9af70-a2f1-4a22-a229-717596a4c6e7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56092025-04-30T06:35:44.407ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
56102025-04-30T06:35:44.410ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f1728675-da7c-4917-a194-a63cf814b9ae", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56112025-04-30T06:35:44.410ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
56122025-04-30T06:35:44.410ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77bad771-5870-4b00-b5e4-4649f152ad33", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56132025-04-30T06:35:44.410ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
56142025-04-30T06:35:44.411ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "58627d3a-806f-4b71-b876-afa3e2d51e4f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56152025-04-30T06:35:44.411ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
56162025-04-30T06:35:44.416ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5fad330e-950a-488a-9b04-609b25c302ca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56172025-04-30T06:35:44.416ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
56182025-04-30T06:35:44.417ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56192025-04-30T06:35:44.417ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56202025-04-30T06:35:44.417ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56212025-04-30T06:35:44.417ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56222025-04-30T06:35:44.429ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56232025-04-30T06:35:44.433ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "86067bd2-2363-4f04-a241-ca1813649c84", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56242025-04-30T06:35:44.434ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
56252025-04-30T06:35:44.439ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56262025-04-30T06:35:44.449ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56272025-04-30T06:35:44.449ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56282025-04-30T06:35:44.450ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56292025-04-30T06:35:44.451ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56302025-04-30T06:35:44.455ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56312025-04-30T06:35:44.455ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56322025-04-30T06:35:44.456ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56332025-04-30T06:35:44.457ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "27bf56e0-435a-4d50-985b-b43ab9a78836", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56342025-04-30T06:35:44.457ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
56352025-04-30T06:35:44.465ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "98e23a35-5784-4a47-a200-bb331bd33291", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56362025-04-30T06:35:44.466ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
56372025-04-30T06:35:44.470ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cee22694-7dd7-4486-8f30-37ec9f8379ec", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56382025-04-30T06:35:44.470ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
56392025-04-30T06:35:44.473ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aba7adcc-6dd0-44eb-925d-e3c53f318b31", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56402025-04-30T06:35:44.474ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
56412025-04-30T06:35:44.480ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8084f446-a9a9-4f21-b686-a8f8ce1441c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56422025-04-30T06:35:44.480ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
56432025-04-30T06:35:44.483ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd8e8705-42fa-4b35-aaef-a9433a4f263d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56442025-04-30T06:35:44.483ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
56452025-04-30T06:35:44.485ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb27be65-6906-4dba-a7d7-f10af60042d3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56462025-04-30T06:35:44.485ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
56472025-04-30T06:35:44.487ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4cafc6f9-6362-4322-8b16-db66eee5fb1d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56482025-04-30T06:35:44.487ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
56492025-04-30T06:35:44.490ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "97d6ee0a-831b-4fa2-9a05-6ee1738251fa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56502025-04-30T06:35:44.490ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
56512025-04-30T06:35:44.491ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
56522025-04-30T06:35:44.494ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2c8e105-b9eb-440e-afcb-f69036af7cca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56532025-04-30T06:35:44.494ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
56542025-04-30T06:35:44.495ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dae1d26-56a4-43fb-9e58-4b8700d1cf23", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56552025-04-30T06:35:44.495ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
56562025-04-30T06:35:44.497ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "939c3824-acee-4484-9229-c2219b3f3f4a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56572025-04-30T06:35:44.497ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
56582025-04-30T06:35:44.499ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f96328-c8b0-4202-9d3a-05a86542e43a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56592025-04-30T06:35:44.499ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
56602025-04-30T06:35:44.506ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a03600e-5e61-4856-a48c-22ef788b1eae", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56612025-04-30T06:35:44.506ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
56622025-04-30T06:35:44.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e7c0b50-b4e4-402e-bc1e-3f4ff1b1aa1c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:43 GMT"} })
56632025-04-30T06:35:44.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
56642025-04-30T06:35:44.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56652025-04-30T06:35:44.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56662025-04-30T06:35:44.515ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56672025-04-30T06:35:44.516ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56682025-04-30T06:35:44.516ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56692025-04-30T06:35:44.516ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
56702025-04-30T06:35:44.516ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b9863a2-fd65-40dd-be4a-10f51f387050", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56712025-04-30T06:35:44.516ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
56722025-04-30T06:35:44.518ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a5e37af-e136-46ba-8338-6c19680f2284", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56732025-04-30T06:35:44.518ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
56742025-04-30T06:35:44.519ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d96cdaf-794c-4086-8d89-4c77e9bb17c6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56752025-04-30T06:35:44.519ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
56762025-04-30T06:35:44.521ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "70fecbd3-1128-4bf1-93fa-ce2971d63a90", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56772025-04-30T06:35:44.521ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
56782025-04-30T06:35:44.523ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b575f9c4-cb94-4cee-88c2-b217969c5d36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56792025-04-30T06:35:44.525ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
56802025-04-30T06:35:44.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd14bc51-b6d0-4a54-b9ab-abe5e6156a48", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56812025-04-30T06:35:44.528ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
56822025-04-30T06:35:44.531ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "747814ce-7551-4a1d-8eda-b6f2e2ea0c87", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56832025-04-30T06:35:44.532ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
56842025-04-30T06:35:44.535ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "600fc7df-7f59-4c94-a50c-eedfcc08ad0a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56852025-04-30T06:35:44.535ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
56862025-04-30T06:35:44.540ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "66311f24-61b6-43cb-ba00-30a9ffffd705", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56872025-04-30T06:35:44.540ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
56882025-04-30T06:35:44.542ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
56892025-04-30T06:35:44.542ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
56902025-04-30T06:35:44.542ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56912025-04-30T06:35:44.542ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56922025-04-30T06:35:44.543ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
56932025-04-30T06:35:44.545ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59d10f2f-b58c-4964-a2b7-10f1bd001fd4", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
56942025-04-30T06:35:44.545ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
56952025-04-30T06:35:44.545ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
56962025-04-30T06:35:44.554ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
56972025-04-30T06:35:44.554ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
56982025-04-30T06:35:44.554ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
56992025-04-30T06:35:44.555ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57002025-04-30T06:35:44.565ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57012025-04-30T06:35:44.565ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57022025-04-30T06:35:44.566ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57032025-04-30T06:35:44.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57042025-04-30T06:35:44.567ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57052025-04-30T06:35:44.567ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
57062025-04-30T06:35:44.568ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
57072025-04-30T06:35:44.571ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c56053a-321e-432e-871a-7c7ebbf3cafc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57082025-04-30T06:35:44.571ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
57092025-04-30T06:35:44.573ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
57102025-04-30T06:35:44.573ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
57112025-04-30T06:35:44.578ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae64e135-6f44-4dd0-b1c0-5569213028b9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57122025-04-30T06:35:44.579ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
57132025-04-30T06:35:44.594ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
57142025-04-30T06:35:44.594ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
57152025-04-30T06:35:44.594ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c967f3e-bbdc-4599-b2fb-e618f7f70a31", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57162025-04-30T06:35:44.594ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
57172025-04-30T06:35:44.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18be5076-46f5-43e0-a121-aae64a6c4d1a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57182025-04-30T06:35:44.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
57192025-04-30T06:35:44.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9db266a-3661-4ffe-a88b-2edc50a8f0bc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57202025-04-30T06:35:44.597ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
57212025-04-30T06:35:44.599ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a95f78d9-c7b2-446b-9690-563f01388eb8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57222025-04-30T06:35:44.599ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
57232025-04-30T06:35:44.601ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "175a172f-2e8e-48ba-b4ab-139b52f64ab1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57242025-04-30T06:35:44.601ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57252025-04-30T06:35:44.601ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57262025-04-30T06:35:44.601ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57272025-04-30T06:35:44.601ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57282025-04-30T06:35:44.602ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57292025-04-30T06:35:44.606ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57302025-04-30T06:35:44.610ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57312025-04-30T06:35:44.610ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
57322025-04-30T06:35:44.610ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57332025-04-30T06:35:44.610ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57342025-04-30T06:35:44.610ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57352025-04-30T06:35:44.611ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57362025-04-30T06:35:44.617ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
57372025-04-30T06:35:44.617ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57382025-04-30T06:35:44.621ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
57392025-04-30T06:35:44.621ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57402025-04-30T06:35:44.621ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57412025-04-30T06:35:44.622ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57422025-04-30T06:35:44.681ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
57432025-04-30T06:35:44.681ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
57442025-04-30T06:35:44.681ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57452025-04-30T06:35:44.681ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57462025-04-30T06:35:44.682ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57472025-04-30T06:35:44.682ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57482025-04-30T06:35:44.682ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57492025-04-30T06:35:44.683ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:32968 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
57502025-04-30T06:35:44.683ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
57512025-04-30T06:35:44.683ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:32968 req_id = d74ac50d-28bb-44e5-a76e-98b1109ca71d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57522025-04-30T06:35:44.684ZINFOtest_metrics (dropshot): request completed latency_us = 1137 local_addr = [::1]:56237 method = GET remote_addr = [::1]:32968 req_id = d74ac50d-28bb-44e5-a76e-98b1109ca71d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
57532025-04-30T06:35:44.685ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "d74ac50d-28bb-44e5-a76e-98b1109ca71d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
57542025-04-30T06:35:44.688ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57552025-04-30T06:35:44.688ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
57562025-04-30T06:35:44.688ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
57572025-04-30T06:35:44.691ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57582025-04-30T06:35:44.691ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57592025-04-30T06:35:44.691ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57602025-04-30T06:35:44.691ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57612025-04-30T06:35:44.692ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57622025-04-30T06:35:44.693ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57632025-04-30T06:35:44.699ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57642025-04-30T06:35:44.699ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57652025-04-30T06:35:44.699ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57662025-04-30T06:35:44.699ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57672025-04-30T06:35:44.701ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57682025-04-30T06:35:44.713ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57692025-04-30T06:35:44.713ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57702025-04-30T06:35:44.713ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57712025-04-30T06:35:44.714ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57722025-04-30T06:35:44.718ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57732025-04-30T06:35:44.718ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57742025-04-30T06:35:44.719ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57752025-04-30T06:35:44.764ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
57762025-04-30T06:35:44.813ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
57772025-04-30T06:35:44.813ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
57782025-04-30T06:35:44.813ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 631.931884ms iteration = 14 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
57792025-04-30T06:35:44.814ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 15 reason = Signaled
57802025-04-30T06:35:44.814ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
57812025-04-30T06:35:44.814ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57822025-04-30T06:35:44.814ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57832025-04-30T06:35:44.814ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57842025-04-30T06:35:44.815ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57852025-04-30T06:35:44.815ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
57862025-04-30T06:35:44.816ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
57872025-04-30T06:35:44.816ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57882025-04-30T06:35:44.816ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57892025-04-30T06:35:44.816ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
57902025-04-30T06:35:44.823ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57912025-04-30T06:35:44.827ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
57922025-04-30T06:35:44.827ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
57932025-04-30T06:35:44.827ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
57942025-04-30T06:35:44.828ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
57952025-04-30T06:35:44.831ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
57962025-04-30T06:35:44.837ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
57972025-04-30T06:35:44.837ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
57982025-04-30T06:35:44.837ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
57992025-04-30T06:35:44.837ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58002025-04-30T06:35:44.838ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
58012025-04-30T06:35:44.838ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
58022025-04-30T06:35:44.839ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58032025-04-30T06:35:44.840ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58042025-04-30T06:35:44.840ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58052025-04-30T06:35:44.841ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: aa 0e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61570 req_id = 9733d0eb-bd12-4776-b82a-cc679ec78292
58062025-04-30T06:35:44.841ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43534,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61570 req_id = 9733d0eb-bd12-4776-b82a-cc679ec78292
58072025-04-30T06:35:44.841ZDEBGtest_metrics (store): query key key = _mgs._tcp
58082025-04-30T06:35:44.842ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58092025-04-30T06:35:44.842ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61570 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9733d0eb-bd12-4776-b82a-cc679ec78292
58102025-04-30T06:35:44.842ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3809, tv_nsec: 641782301 } })
58112025-04-30T06:35:44.843ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
58122025-04-30T06:35:44.843ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5d cc 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ]...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55146 req_id = a273c647-4f1e-40a4-9713-994bda94c969
58132025-04-30T06:35:44.843ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24012,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55146 req_id = a273c647-4f1e-40a4-9713-994bda94c969
58142025-04-30T06:35:44.844ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
58152025-04-30T06:35:44.844ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55146 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a273c647-4f1e-40a4-9713-994bda94c969
58162025-04-30T06:35:44.904ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
58172025-04-30T06:35:44.907ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c6 8e 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51892 req_id = b93ad49a-3775-483e-a78e-f2db9e698074
58182025-04-30T06:35:44.907ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50830,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51892 req_id = b93ad49a-3775-483e-a78e-f2db9e698074
58192025-04-30T06:35:44.907ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
58202025-04-30T06:35:44.908ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50830, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51892 req_id = b93ad49a-3775-483e-a78e-f2db9e698074
58212025-04-30T06:35:44.908ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
58222025-04-30T06:35:44.908ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
58232025-04-30T06:35:44.908ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
58242025-04-30T06:35:44.911ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "639b14a0-2329-4d5d-8967-eb3d0a00d480", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58252025-04-30T06:35:44.912ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
58262025-04-30T06:35:44.915ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b243876-024f-4647-b7b8-c39cad64f87c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58272025-04-30T06:35:44.915ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
58282025-04-30T06:35:44.919ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cc1ae3a-225a-4ebe-9549-408253feeced", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58292025-04-30T06:35:44.919ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
58302025-04-30T06:35:44.922ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad3ce1c0-9585-40f3-8620-83f0f14eaaa9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58312025-04-30T06:35:44.922ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
58322025-04-30T06:35:44.928ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a26fd76-b798-4e7b-b0c6-1fe018eb5815", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58332025-04-30T06:35:44.928ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
58342025-04-30T06:35:44.934ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "666180ba-1d79-4b3c-b38b-e7b1c5de15c6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58352025-04-30T06:35:44.934ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
58362025-04-30T06:35:44.935ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
58372025-04-30T06:35:44.937ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "730aa2a6-f768-4339-8570-e59dc63ed963", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58382025-04-30T06:35:44.937ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
58392025-04-30T06:35:44.942ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45a084f1-7fd3-4817-9f6b-c9e47fd78559", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58402025-04-30T06:35:44.942ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
58412025-04-30T06:35:44.943ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02c6681f-ebc9-4cee-bea8-b88b0198df34", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58422025-04-30T06:35:44.943ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
58432025-04-30T06:35:44.944ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f2d53e4-5743-4e50-b607-afc26d04619e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58442025-04-30T06:35:44.945ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
58452025-04-30T06:35:44.945ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f755a4ee-e943-447f-ba96-05cf66d641a3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58462025-04-30T06:35:44.946ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
58472025-04-30T06:35:44.947ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9b16169-0096-42d6-b2f4-57b8e949b79d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58482025-04-30T06:35:44.947ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
58492025-04-30T06:35:44.948ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f58a8ded-2075-4d77-8374-6b6bed78e612", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58502025-04-30T06:35:44.949ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
58512025-04-30T06:35:44.952ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9298822f-4bd9-4987-86dc-ae34aea91546", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58522025-04-30T06:35:44.952ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
58532025-04-30T06:35:44.954ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be5e08be-f1a0-48a6-a99a-8f0703ea127e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58542025-04-30T06:35:44.954ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
58552025-04-30T06:35:44.956ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "34020f3c-fc2d-4ab8-a328-971fcda7186d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58562025-04-30T06:35:44.956ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
58572025-04-30T06:35:44.958ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72ff2c16-c061-472d-a77c-754c9660f806", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58582025-04-30T06:35:44.958ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
58592025-04-30T06:35:44.960ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfd39d75-c02e-44dd-ac23-b44499eb273a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58602025-04-30T06:35:44.960ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
58612025-04-30T06:35:44.964ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cb8dac9-896b-468f-aa43-d2fc9c6b1ea5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58622025-04-30T06:35:44.964ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
58632025-04-30T06:35:44.965ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5164e443-17e2-4a99-b0ff-92a3f07f947c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58642025-04-30T06:35:44.965ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
58652025-04-30T06:35:44.967ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff1cc8ee-f390-495e-b818-db955d9a49f4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58662025-04-30T06:35:44.967ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
58672025-04-30T06:35:44.968ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "015571ed-452e-4e15-a6d8-ef1d577d62a2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58682025-04-30T06:35:44.969ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
58692025-04-30T06:35:44.970ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6be0caea-3ab7-40a8-8baa-fc30b99304ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58702025-04-30T06:35:44.970ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
58712025-04-30T06:35:44.972ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf79b0eb-c95b-48b6-a064-ddf4b9c3b3ba", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58722025-04-30T06:35:44.972ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
58732025-04-30T06:35:44.974ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "659e93b5-8d54-49c5-894d-662d3b2d558f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58742025-04-30T06:35:44.974ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
58752025-04-30T06:35:44.976ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95bca79d-ae92-4c60-a4cb-13045e5bacc4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58762025-04-30T06:35:44.976ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
58772025-04-30T06:35:44.978ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a2be222-586f-40c0-895f-f9ed2c7a8c26", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58782025-04-30T06:35:44.979ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
58792025-04-30T06:35:44.982ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6d9126f-8b05-47d5-a00a-21e2e43d5daa", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58802025-04-30T06:35:44.982ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
58812025-04-30T06:35:44.984ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "147dfa49-1d75-4e34-b963-083e4ab1ae0b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58822025-04-30T06:35:44.985ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
58832025-04-30T06:35:44.986ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
58842025-04-30T06:35:44.986ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
58852025-04-30T06:35:44.986ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58862025-04-30T06:35:44.986ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58872025-04-30T06:35:44.986ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58882025-04-30T06:35:44.988ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "553ffc6b-1213-41cb-aa3a-0bded34d5c0f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58892025-04-30T06:35:44.988ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
58902025-04-30T06:35:44.989ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
58912025-04-30T06:35:44.994ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
58922025-04-30T06:35:44.994ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58932025-04-30T06:35:44.994ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58942025-04-30T06:35:44.994ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
58952025-04-30T06:35:44.995ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9119b73a-0944-4c84-9e41-4253793a5be6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
58962025-04-30T06:35:44.995ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
58972025-04-30T06:35:44.998ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
58982025-04-30T06:35:44.998ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
58992025-04-30T06:35:44.999ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59002025-04-30T06:35:44.999ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "447175d3-7f8a-4696-8578-c7ffa92647ed", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59012025-04-30T06:35:45.000ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
59022025-04-30T06:35:45.005ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c22266be-9361-46be-be84-d117e72e5980", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59032025-04-30T06:35:45.006ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
59042025-04-30T06:35:45.009ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2576d763-cd14-489e-b1b5-58b0ac9208e3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59052025-04-30T06:35:45.009ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
59062025-04-30T06:35:45.011ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc709c11-a138-4d1c-bd1a-3badb13b6ded", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59072025-04-30T06:35:45.012ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
59082025-04-30T06:35:45.023ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d615f3d-2809-4bd4-a43f-73f4f3f9a118", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59092025-04-30T06:35:45.023ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
59102025-04-30T06:35:45.027ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "415c6ca5-c4d3-4977-9049-c5db5a5a942e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59112025-04-30T06:35:45.027ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
59122025-04-30T06:35:45.032ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6bef614f-be99-42bf-8da3-c02e209f8376", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59132025-04-30T06:35:45.033ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
59142025-04-30T06:35:45.036ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2de3521b-5d77-4863-b4b7-b00d5a6b0a64", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59152025-04-30T06:35:45.036ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
59162025-04-30T06:35:45.039ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dda5823-b286-4ebf-84db-79336ec5a925", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59172025-04-30T06:35:45.039ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
59182025-04-30T06:35:45.044ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b062435a-0df4-4dd7-afdf-cfb54a088de3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59192025-04-30T06:35:45.044ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
59202025-04-30T06:35:45.044ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
59212025-04-30T06:35:45.046ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c2e01d-bfce-4e7a-8d0c-baa666eef889", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59222025-04-30T06:35:45.046ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
59232025-04-30T06:35:45.049ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86b9c010-730e-4bc1-8b5d-989c90ccfa56", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59242025-04-30T06:35:45.049ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
59252025-04-30T06:35:45.050ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "20952053-42b7-4886-b656-c24e7c3c8172", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59262025-04-30T06:35:45.050ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
59272025-04-30T06:35:45.052ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff636d2e-241b-4cde-a67d-576e8ec33c3e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59282025-04-30T06:35:45.052ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59292025-04-30T06:35:45.052ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59302025-04-30T06:35:45.052ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59312025-04-30T06:35:45.052ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59322025-04-30T06:35:45.054ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59332025-04-30T06:35:45.057ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59342025-04-30T06:35:45.062ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59352025-04-30T06:35:45.062ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
59362025-04-30T06:35:45.062ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59372025-04-30T06:35:45.062ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59382025-04-30T06:35:45.062ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59392025-04-30T06:35:45.062ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59402025-04-30T06:35:45.065ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59412025-04-30T06:35:45.071ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
59422025-04-30T06:35:45.071ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59432025-04-30T06:35:45.072ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59442025-04-30T06:35:45.072ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59452025-04-30T06:35:45.166ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
59462025-04-30T06:35:45.166ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
59472025-04-30T06:35:45.166ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
59482025-04-30T06:35:45.166ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59492025-04-30T06:35:45.166ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59502025-04-30T06:35:45.166ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59512025-04-30T06:35:45.167ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59522025-04-30T06:35:45.167ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:48148 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
59532025-04-30T06:35:45.168ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:48148 req_id = a77eee7c-e78e-46d4-90eb-b3484450e437 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59542025-04-30T06:35:45.169ZINFOtest_metrics (dropshot): request completed latency_us = 740 local_addr = [::1]:56237 method = GET remote_addr = [::1]:48148 req_id = a77eee7c-e78e-46d4-90eb-b3484450e437 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
59552025-04-30T06:35:45.169ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a77eee7c-e78e-46d4-90eb-b3484450e437", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
59562025-04-30T06:35:45.172ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59572025-04-30T06:35:45.172ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
59582025-04-30T06:35:45.172ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
59592025-04-30T06:35:45.174ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59602025-04-30T06:35:45.174ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59612025-04-30T06:35:45.174ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59622025-04-30T06:35:45.174ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59632025-04-30T06:35:45.175ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59642025-04-30T06:35:45.175ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59652025-04-30T06:35:45.180ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
59662025-04-30T06:35:45.180ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59672025-04-30T06:35:45.180ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59682025-04-30T06:35:45.180ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59692025-04-30T06:35:45.182ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59702025-04-30T06:35:45.188ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59712025-04-30T06:35:45.188ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59722025-04-30T06:35:45.188ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59732025-04-30T06:35:45.189ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59742025-04-30T06:35:45.189ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
59752025-04-30T06:35:45.189ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
59762025-04-30T06:35:45.190ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
59772025-04-30T06:35:45.216ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
59782025-04-30T06:35:45.216ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
59792025-04-30T06:35:45.216ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 402.70853ms iteration = 15 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
59802025-04-30T06:35:45.216ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 16 reason = Signaled
59812025-04-30T06:35:45.216ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
59822025-04-30T06:35:45.216ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
59832025-04-30T06:35:45.216ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59842025-04-30T06:35:45.216ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59852025-04-30T06:35:45.217ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59862025-04-30T06:35:45.219ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
59872025-04-30T06:35:45.224ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
59882025-04-30T06:35:45.224ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
59892025-04-30T06:35:45.225ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
59902025-04-30T06:35:45.225ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
59912025-04-30T06:35:45.231ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
59922025-04-30T06:35:45.231ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
59932025-04-30T06:35:45.232ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a4 36 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .6..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59242 req_id = e0cb2ac0-7682-4079-89af-1f04a413667f
59942025-04-30T06:35:45.232ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42038,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59242 req_id = e0cb2ac0-7682-4079-89af-1f04a413667f
59952025-04-30T06:35:45.232ZDEBGtest_metrics (store): query key key = _mgs._tcp
59962025-04-30T06:35:45.232ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
59972025-04-30T06:35:45.233ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59242 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e0cb2ac0-7682-4079-89af-1f04a413667f
59982025-04-30T06:35:45.234ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3810, tv_nsec: 33740021 } })
59992025-04-30T06:35:45.234ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
60002025-04-30T06:35:45.235ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd ef 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60606 req_id = a05cd1f1-559a-4e42-8fdb-b47f70839043
60012025-04-30T06:35:45.235ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48623,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60606 req_id = a05cd1f1-559a-4e42-8fdb-b47f70839043
60022025-04-30T06:35:45.235ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
60032025-04-30T06:35:45.235ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60606 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a05cd1f1-559a-4e42-8fdb-b47f70839043
60042025-04-30T06:35:45.308ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
60052025-04-30T06:35:45.309ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
60062025-04-30T06:35:45.309ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60072025-04-30T06:35:45.313ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 25fcd7e7-02d4-4ef8-9220-194322a244bc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60082025-04-30T06:35:45.313ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 22 c0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c "............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:65205 req_id = ef779989-4e70-4a06-b24d-635dae60548e
60092025-04-30T06:35:45.313ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 8896,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65205 req_id = ef779989-4e70-4a06-b24d-635dae60548e
60102025-04-30T06:35:45.313ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
60112025-04-30T06:35:45.313ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 8896, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:65205 req_id = ef779989-4e70-4a06-b24d-635dae60548e
60122025-04-30T06:35:45.314ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 25fcd7e7-02d4-4ef8-9220-194322a244bc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60132025-04-30T06:35:45.314ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 25fcd7e7-02d4-4ef8-9220-194322a244bc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60142025-04-30T06:35:45.315ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 25fcd7e7-02d4-4ef8-9220-194322a244bc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60152025-04-30T06:35:45.318ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
60162025-04-30T06:35:45.318ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
60172025-04-30T06:35:45.318ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
60182025-04-30T06:35:45.320ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
60192025-04-30T06:35:45.320ZINFOtest_metrics (dropshot_internal): request completed latency_us = 7954 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 25fcd7e7-02d4-4ef8-9220-194322a244bc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
60202025-04-30T06:35:45.323ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "25fcd7e7-02d4-4ef8-9220-194322a244bc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60212025-04-30T06:35:45.323ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60222025-04-30T06:35:45.324ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 507460c0-661a-4d2a-964b-13ea2305e4da uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60232025-04-30T06:35:45.330ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 507460c0-661a-4d2a-964b-13ea2305e4da resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60242025-04-30T06:35:45.331ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 507460c0-661a-4d2a-964b-13ea2305e4da roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60252025-04-30T06:35:45.332ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 507460c0-661a-4d2a-964b-13ea2305e4da resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60262025-04-30T06:35:45.332ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "b880439e-0ddb-4e5c-9c7c-5e74dcfc6098", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60272025-04-30T06:35:45.333ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
60282025-04-30T06:35:45.337ZINFOtest_metrics (dropshot_internal): request completed latency_us = 12780 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 507460c0-661a-4d2a-964b-13ea2305e4da response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
60292025-04-30T06:35:45.340ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "507460c0-661a-4d2a-964b-13ea2305e4da", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60302025-04-30T06:35:45.340ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
60312025-04-30T06:35:45.340ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
60322025-04-30T06:35:45.340ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
60332025-04-30T06:35:45.347ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37b79dbe-d38b-450a-b582-e1d325a8c315", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60342025-04-30T06:35:45.348ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
60352025-04-30T06:35:45.385ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
60362025-04-30T06:35:45.385ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
60372025-04-30T06:35:45.385ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60382025-04-30T06:35:45.385ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60392025-04-30T06:35:45.388ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60402025-04-30T06:35:45.398ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
60412025-04-30T06:35:45.405ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
60422025-04-30T06:35:45.405ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60432025-04-30T06:35:45.405ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60442025-04-30T06:35:45.405ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60452025-04-30T06:35:45.408ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
60462025-04-30T06:35:45.408ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
60472025-04-30T06:35:45.409ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
60482025-04-30T06:35:45.409ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c87f8a22-aeb8-4a49-8696-100310b2dfac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60492025-04-30T06:35:45.409ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
60502025-04-30T06:35:45.413ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b7725f1-557c-4b0b-b4ac-8511e0951a2c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60512025-04-30T06:35:45.413ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
60522025-04-30T06:35:45.416ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab227b06-c4d6-498d-a1e6-29081ce4f968", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60532025-04-30T06:35:45.416ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
60542025-04-30T06:35:45.419ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd043a6e-4a1a-4982-b4ec-e99fe44f84d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60552025-04-30T06:35:45.419ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
60562025-04-30T06:35:45.422ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dde6626e-908f-4a71-bdac-235b53cce09c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60572025-04-30T06:35:45.422ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
60582025-04-30T06:35:45.429ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14e1b2fb-5ebf-4d31-a063-36b6e6323c80", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60592025-04-30T06:35:45.429ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
60602025-04-30T06:35:45.432ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9a3cf0a-9aec-42e7-aebb-df79d2aa3124", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60612025-04-30T06:35:45.432ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
60622025-04-30T06:35:45.434ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86fcf987-aaed-43ba-a8ed-31e575100a07", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60632025-04-30T06:35:45.434ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
60642025-04-30T06:35:45.440ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
60652025-04-30T06:35:45.441ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "122eeca4-c15b-44a7-950a-ae9aa20d554e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60662025-04-30T06:35:45.441ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
60672025-04-30T06:35:45.443ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a22c9da9-eef5-4339-ab36-fcf36502601e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60682025-04-30T06:35:45.443ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
60692025-04-30T06:35:45.445ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99106edf-1f09-412b-9e31-428863ede947", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60702025-04-30T06:35:45.445ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
60712025-04-30T06:35:45.449ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe54b571-0bbf-44cc-b013-de3f4dd91463", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60722025-04-30T06:35:45.450ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
60732025-04-30T06:35:45.452ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6c1151d-1c2f-43cb-bf9a-21dd66f2880c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60742025-04-30T06:35:45.452ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
60752025-04-30T06:35:45.455ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03ebbbcf-60a9-407a-8d5a-b94f85619afb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60762025-04-30T06:35:45.455ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
60772025-04-30T06:35:45.457ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fea9d38-5d2a-4023-ab3e-c9073c5d679d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60782025-04-30T06:35:45.457ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
60792025-04-30T06:35:45.460ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd9a6f29-47ba-45b5-947d-6e2aa4a9a58b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60802025-04-30T06:35:45.460ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
60812025-04-30T06:35:45.462ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0819386-c1d9-49a1-8b62-527a8e46da57", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60822025-04-30T06:35:45.462ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
60832025-04-30T06:35:45.463ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "522724e2-9168-46fa-a3c8-8a0738850144", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60842025-04-30T06:35:45.463ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
60852025-04-30T06:35:45.464ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "135d9ecf-73bf-4aaf-957d-84165baabe04", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60862025-04-30T06:35:45.464ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
60872025-04-30T06:35:45.465ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54c1ac2-c977-4316-b4e8-d75d54622a06", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60882025-04-30T06:35:45.465ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
60892025-04-30T06:35:45.466ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b78b6ab-a005-4071-b4cc-bc8c4c0a855d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60902025-04-30T06:35:45.466ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
60912025-04-30T06:35:45.467ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e033e774-9d88-4200-ba54-1eaa1bc9c678", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60922025-04-30T06:35:45.467ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
60932025-04-30T06:35:45.471ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9d7f2b8-f227-4f89-b24f-f2ae2c421e9b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60942025-04-30T06:35:45.471ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
60952025-04-30T06:35:45.473ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "214b521e-16f6-469f-9f8a-d67ff959cb23", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60962025-04-30T06:35:45.473ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
60972025-04-30T06:35:45.485ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8fa8df7-3ace-4094-a662-c16688e61034", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
60982025-04-30T06:35:45.485ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
60992025-04-30T06:35:45.487ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "720a4819-5082-49d1-846e-58c93e59c6c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61002025-04-30T06:35:45.488ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
61012025-04-30T06:35:45.490ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1c055747-a6b3-4cb4-bf01-19e0aa5d9758", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61022025-04-30T06:35:45.491ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
61032025-04-30T06:35:45.491ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61042025-04-30T06:35:45.491ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61052025-04-30T06:35:45.491ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61062025-04-30T06:35:45.491ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61072025-04-30T06:35:45.492ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61082025-04-30T06:35:45.494ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "724c812a-29f1-44ab-8fd4-8d311a335836", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61092025-04-30T06:35:45.495ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
61102025-04-30T06:35:45.495ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61112025-04-30T06:35:45.503ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
61122025-04-30T06:35:45.503ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61132025-04-30T06:35:45.503ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61142025-04-30T06:35:45.503ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61152025-04-30T06:35:45.505ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d80d895e-81f4-426f-a07c-72ebcad08471", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61162025-04-30T06:35:45.505ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
61172025-04-30T06:35:45.506ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61182025-04-30T06:35:45.506ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61192025-04-30T06:35:45.506ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61202025-04-30T06:35:45.507ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9547e678-b310-4bf1-ad78-5541f581a7b2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61212025-04-30T06:35:45.507ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
61222025-04-30T06:35:45.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "535b2029-a230-4667-88ac-4efd2c343f8d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61232025-04-30T06:35:45.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
61242025-04-30T06:35:45.510ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ff10de5-2287-4b40-9ebd-29db9eaadb35", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61252025-04-30T06:35:45.510ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
61262025-04-30T06:35:45.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61272025-04-30T06:35:45.512ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e729ced5-a2c2-4738-ad91-3aa947c3ad1a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:44 GMT"} })
61282025-04-30T06:35:45.512ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
61292025-04-30T06:35:45.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61302025-04-30T06:35:45.516ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61312025-04-30T06:35:45.518ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61322025-04-30T06:35:45.518ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61332025-04-30T06:35:45.518ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
61342025-04-30T06:35:45.522ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bf906b6-d607-40be-baa4-f880a5786392", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61352025-04-30T06:35:45.522ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
61362025-04-30T06:35:45.525ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f628fb42-3adc-49a3-b119-de98cbc2a355", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61372025-04-30T06:35:45.525ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
61382025-04-30T06:35:45.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3a3999d-368c-4974-97da-c91c52510363", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61392025-04-30T06:35:45.527ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
61402025-04-30T06:35:45.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9182e38e-914a-4987-b9e4-a54a6a393991", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61412025-04-30T06:35:45.530ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
61422025-04-30T06:35:45.533ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee5515eb-1ee2-4f1b-a583-559ee5123f57", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61432025-04-30T06:35:45.533ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
61442025-04-30T06:35:45.536ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "45a06a6b-86d6-46bb-9a55-febf3e00428a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61452025-04-30T06:35:45.537ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
61462025-04-30T06:35:45.537ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
61472025-04-30T06:35:45.538ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6a43bb3-ac15-4f89-8881-96c8f5de1676", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61482025-04-30T06:35:45.538ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
61492025-04-30T06:35:45.539ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c10f56bd-1e44-48e0-a764-31d486b1e626", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61502025-04-30T06:35:45.539ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
61512025-04-30T06:35:45.540ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dca4dcad-0eb7-4fe5-a796-50cfffaf344d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61522025-04-30T06:35:45.540ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
61532025-04-30T06:35:45.541ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42c71252-f312-4e97-a29d-20ea19281575", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61542025-04-30T06:35:45.541ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61552025-04-30T06:35:45.541ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61562025-04-30T06:35:45.541ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61572025-04-30T06:35:45.541ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61582025-04-30T06:35:45.542ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61592025-04-30T06:35:45.549ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61602025-04-30T06:35:45.558ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61612025-04-30T06:35:45.558ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
61622025-04-30T06:35:45.558ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61632025-04-30T06:35:45.558ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61642025-04-30T06:35:45.558ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61652025-04-30T06:35:45.559ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61662025-04-30T06:35:45.563ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61672025-04-30T06:35:45.568ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
61682025-04-30T06:35:45.568ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61692025-04-30T06:35:45.569ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61702025-04-30T06:35:45.570ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61712025-04-30T06:35:45.570ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61722025-04-30T06:35:45.570ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61732025-04-30T06:35:45.570ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
61742025-04-30T06:35:45.571ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
61752025-04-30T06:35:45.575ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
61762025-04-30T06:35:45.575ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
61772025-04-30T06:35:45.628ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
61782025-04-30T06:35:45.628ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
61792025-04-30T06:35:45.629ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
61802025-04-30T06:35:45.629ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61812025-04-30T06:35:45.629ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
61822025-04-30T06:35:45.629ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
61832025-04-30T06:35:45.630ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
61842025-04-30T06:35:45.630ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:64523 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
61852025-04-30T06:35:45.633ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
61862025-04-30T06:35:45.633ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
61872025-04-30T06:35:45.633ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:64523 req_id = 0515872d-8d74-44ed-b0e9-9f8b9ff7833a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61882025-04-30T06:35:45.634ZINFOtest_metrics (dropshot): request completed latency_us = 558 local_addr = [::1]:56237 method = GET remote_addr = [::1]:64523 req_id = 0515872d-8d74-44ed-b0e9-9f8b9ff7833a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
61892025-04-30T06:35:45.634ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0515872d-8d74-44ed-b0e9-9f8b9ff7833a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
61902025-04-30T06:35:45.637ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61912025-04-30T06:35:45.637ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
61922025-04-30T06:35:45.637ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
61932025-04-30T06:35:45.639ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
61942025-04-30T06:35:45.639ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
61952025-04-30T06:35:45.639ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
61962025-04-30T06:35:45.639ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
61972025-04-30T06:35:45.647ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
61982025-04-30T06:35:45.648ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
61992025-04-30T06:35:45.654ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62002025-04-30T06:35:45.654ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62012025-04-30T06:35:45.654ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62022025-04-30T06:35:45.655ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62032025-04-30T06:35:45.657ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62042025-04-30T06:35:45.663ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62052025-04-30T06:35:45.663ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62062025-04-30T06:35:45.663ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62072025-04-30T06:35:45.665ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62082025-04-30T06:35:45.666ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62092025-04-30T06:35:45.666ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62102025-04-30T06:35:45.667ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62112025-04-30T06:35:45.671ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
62122025-04-30T06:35:45.714ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
62132025-04-30T06:35:45.715ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
62142025-04-30T06:35:45.715ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 498.753283ms iteration = 16 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
62152025-04-30T06:35:45.715ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 17 reason = Signaled
62162025-04-30T06:35:45.715ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
62172025-04-30T06:35:45.715ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62182025-04-30T06:35:45.716ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62192025-04-30T06:35:45.716ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62202025-04-30T06:35:45.717ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62212025-04-30T06:35:45.726ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62222025-04-30T06:35:45.732ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
62232025-04-30T06:35:45.732ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
62242025-04-30T06:35:45.732ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
62252025-04-30T06:35:45.733ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
62262025-04-30T06:35:45.734ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
62272025-04-30T06:35:45.737ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
62282025-04-30T06:35:45.738ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
62292025-04-30T06:35:45.738ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8c fa 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56580 req_id = acf5c552-a07c-4b04-999d-e5a13fe658c3
62302025-04-30T06:35:45.738ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36090,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56580 req_id = acf5c552-a07c-4b04-999d-e5a13fe658c3
62312025-04-30T06:35:45.739ZDEBGtest_metrics (store): query key key = _mgs._tcp
62322025-04-30T06:35:45.739ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62332025-04-30T06:35:45.739ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56580 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = acf5c552-a07c-4b04-999d-e5a13fe658c3
62342025-04-30T06:35:45.740ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3810, tv_nsec: 539119407 } })
62352025-04-30T06:35:45.740ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
62362025-04-30T06:35:45.742ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 50 c1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e P...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51543 req_id = d490bdd5-4f4b-4a2c-bd07-63de630de28f
62372025-04-30T06:35:45.742ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20673,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51543 req_id = d490bdd5-4f4b-4a2c-bd07-63de630de28f
62382025-04-30T06:35:45.743ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
62392025-04-30T06:35:45.743ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51543 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d490bdd5-4f4b-4a2c-bd07-63de630de28f
62402025-04-30T06:35:45.801ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
62412025-04-30T06:35:45.801ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
62422025-04-30T06:35:45.802ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
62432025-04-30T06:35:45.802ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62442025-04-30T06:35:45.802ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62452025-04-30T06:35:45.803ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62462025-04-30T06:35:45.803ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 29 12 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c )............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60984 req_id = 37876b30-d5ed-4812-9e2c-9862b1671d37
62472025-04-30T06:35:45.804ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10514,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60984 req_id = 37876b30-d5ed-4812-9e2c-9862b1671d37
62482025-04-30T06:35:45.804ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
62492025-04-30T06:35:45.804ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10514, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60984 req_id = 37876b30-d5ed-4812-9e2c-9862b1671d37
62502025-04-30T06:35:45.805ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
62512025-04-30T06:35:45.805ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
62522025-04-30T06:35:45.805ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
62532025-04-30T06:35:45.807ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
62542025-04-30T06:35:45.815ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
62552025-04-30T06:35:45.815ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62562025-04-30T06:35:45.815ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62572025-04-30T06:35:45.816ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62582025-04-30T06:35:45.820ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9354d76-2091-4e48-b415-15f89c7a8e5d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62592025-04-30T06:35:45.821ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
62602025-04-30T06:35:45.821ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
62612025-04-30T06:35:45.822ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
62622025-04-30T06:35:45.825ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
62632025-04-30T06:35:45.828ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1545f4c3-0d1e-47d7-b522-7721cf8f79f1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62642025-04-30T06:35:45.828ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
62652025-04-30T06:35:45.829ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a5fb8e9-a405-4808-afff-1e7ebfdc05ff", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62662025-04-30T06:35:45.829ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
62672025-04-30T06:35:45.831ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8be67846-5f49-4f12-809b-d921f8454c28", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62682025-04-30T06:35:45.831ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
62692025-04-30T06:35:45.834ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "222e23d9-189a-46db-bbbd-643f3baf7f5b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62702025-04-30T06:35:45.834ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
62712025-04-30T06:35:45.837ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "826bd8b1-2431-4936-9fad-85cd344ea8d4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62722025-04-30T06:35:45.838ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
62732025-04-30T06:35:45.839ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60180229-1593-4c33-a5b8-e20ed61bdcd1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62742025-04-30T06:35:45.839ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
62752025-04-30T06:35:45.845ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e61fbf36-4457-4277-a2d9-1a6aefdfec1a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62762025-04-30T06:35:45.845ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
62772025-04-30T06:35:45.847ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfb1f033-acac-4ac4-bce6-86d0570f952e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62782025-04-30T06:35:45.847ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
62792025-04-30T06:35:45.851ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce379e0f-d219-4e77-9c76-63a2fd2932f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62802025-04-30T06:35:45.851ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
62812025-04-30T06:35:45.856ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ffa99b73-5812-40d7-9044-58387521ea7d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62822025-04-30T06:35:45.856ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
62832025-04-30T06:35:45.856ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
62842025-04-30T06:35:45.858ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f06f3a0-f3bf-46c2-81c5-e58c051b76ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62852025-04-30T06:35:45.858ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
62862025-04-30T06:35:45.860ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2557d99e-f245-4be0-bb0d-69b174d188f2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62872025-04-30T06:35:45.860ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
62882025-04-30T06:35:45.862ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a735c95c-d7f1-476f-95af-f1b3346dbfbe", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62892025-04-30T06:35:45.862ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
62902025-04-30T06:35:45.864ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c78e21db-ee6e-40e5-8883-26ff2e95597f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62912025-04-30T06:35:45.864ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
62922025-04-30T06:35:45.866ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b73a13e9-cab0-48d2-9d93-f3f54ab0c9f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62932025-04-30T06:35:45.866ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
62942025-04-30T06:35:45.867ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79b9ebc2-8fbb-4758-aa9a-b280898dc8bd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62952025-04-30T06:35:45.867ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
62962025-04-30T06:35:45.869ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93dd4673-4d40-4b8b-8360-2f10b73a7411", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62972025-04-30T06:35:45.869ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
62982025-04-30T06:35:45.872ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3778155-b3e7-4802-95e7-0718c631db53", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
62992025-04-30T06:35:45.872ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
63002025-04-30T06:35:45.873ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ad017c3-6bb8-4c4a-b786-7689e31435d8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63012025-04-30T06:35:45.873ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
63022025-04-30T06:35:45.874ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "18e92789-60ee-44cb-b5f0-13f98db7d488", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63032025-04-30T06:35:45.874ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
63042025-04-30T06:35:45.875ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb4305e8-d934-4934-823d-3aee4b1053b5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63052025-04-30T06:35:45.875ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
63062025-04-30T06:35:45.876ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f82b42d3-5762-4190-83ce-455712d021d9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63072025-04-30T06:35:45.876ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
63082025-04-30T06:35:45.877ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51426b57-e81d-4581-a6ee-c5c255fcb505", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63092025-04-30T06:35:45.878ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
63102025-04-30T06:35:45.880ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "850368eb-ebd0-409e-b45d-82f528c205a1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63112025-04-30T06:35:45.880ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
63122025-04-30T06:35:45.883ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a13dc7b2-18b6-47eb-a07e-9d6cc0b35c51", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63132025-04-30T06:35:45.883ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
63142025-04-30T06:35:45.887ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ebfcab6-e993-425e-bd1b-abad241a6038", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63152025-04-30T06:35:45.888ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
63162025-04-30T06:35:45.890ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "65c1599a-eb1a-46d8-bb29-7784f7a006ac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63172025-04-30T06:35:45.890ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
63182025-04-30T06:35:45.892ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a027d88-6199-4494-92d4-9a78ff6e237a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63192025-04-30T06:35:45.893ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
63202025-04-30T06:35:45.895ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca7a64cd-5505-4a28-8064-308d5cf099c4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63212025-04-30T06:35:45.895ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
63222025-04-30T06:35:45.897ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a5cae54-7eea-4ee9-932f-2e933a34b1f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63232025-04-30T06:35:45.897ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
63242025-04-30T06:35:45.898ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "22f69070-11da-4cd1-9aa0-23bed05f70c3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63252025-04-30T06:35:45.900ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
63262025-04-30T06:35:45.903ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a3b85c1-8ea9-49f2-8ea9-ae1b84753b75", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63272025-04-30T06:35:45.903ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
63282025-04-30T06:35:45.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c851832c-ec9f-411d-854a-1a56c04c373b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63292025-04-30T06:35:45.905ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
63302025-04-30T06:35:45.907ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a9ed75d-bcc9-4c42-b11d-8b3dc92dc8a1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63312025-04-30T06:35:45.907ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
63322025-04-30T06:35:45.907ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63332025-04-30T06:35:45.907ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63342025-04-30T06:35:45.908ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63352025-04-30T06:35:45.908ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63362025-04-30T06:35:45.909ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63372025-04-30T06:35:45.911ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4104ac2-1adf-411e-9876-5512de780b2b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63382025-04-30T06:35:45.911ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
63392025-04-30T06:35:45.912ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63402025-04-30T06:35:45.920ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
63412025-04-30T06:35:45.921ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63422025-04-30T06:35:45.921ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63432025-04-30T06:35:45.922ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63442025-04-30T06:35:45.925ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d61ee8cf-c786-4d32-9e94-9a6c82f13a4e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63452025-04-30T06:35:45.925ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
63462025-04-30T06:35:45.926ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63472025-04-30T06:35:45.926ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63482025-04-30T06:35:45.926ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63492025-04-30T06:35:45.929ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "910744dd-2561-4d27-87db-bb34bc46cf2e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63502025-04-30T06:35:45.929ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
63512025-04-30T06:35:45.938ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6be5a9e9-66b2-4b13-ba64-1a9afe3fb201", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63522025-04-30T06:35:45.938ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
63532025-04-30T06:35:45.941ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4428da7e-1635-4a82-b749-331edb770d90", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63542025-04-30T06:35:45.945ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
63552025-04-30T06:35:45.953ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f737014c-3c54-4655-98f7-41d206d3dce6", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63562025-04-30T06:35:45.953ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
63572025-04-30T06:35:45.954ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aeadadc3-2842-476b-90bc-fe259af44501", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63582025-04-30T06:35:45.954ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
63592025-04-30T06:35:45.958ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc513713-110a-4fce-b0f3-fc37d0cae5af", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63602025-04-30T06:35:45.958ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
63612025-04-30T06:35:45.964ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "614daea3-567d-427e-903a-6c5fac089dca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63622025-04-30T06:35:45.964ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
63632025-04-30T06:35:45.967ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a1d4bf0c-c7d1-4b03-9fd1-9a80dcd7d86b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63642025-04-30T06:35:45.968ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63652025-04-30T06:35:45.968ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63662025-04-30T06:35:45.968ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63672025-04-30T06:35:45.968ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63682025-04-30T06:35:45.969ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63692025-04-30T06:35:45.973ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63702025-04-30T06:35:45.977ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63712025-04-30T06:35:45.977ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
63722025-04-30T06:35:45.977ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63732025-04-30T06:35:45.977ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63742025-04-30T06:35:45.977ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63752025-04-30T06:35:45.978ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63762025-04-30T06:35:45.985ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
63772025-04-30T06:35:45.992ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
63782025-04-30T06:35:45.992ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63792025-04-30T06:35:45.992ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
63802025-04-30T06:35:45.993ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
63812025-04-30T06:35:45.994ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
63822025-04-30T06:35:46.077ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
63832025-04-30T06:35:46.077ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
63842025-04-30T06:35:46.078ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
63852025-04-30T06:35:46.078ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63862025-04-30T06:35:46.078ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
63872025-04-30T06:35:46.078ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
63882025-04-30T06:35:46.078ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
63892025-04-30T06:35:46.078ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:43177 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
63902025-04-30T06:35:46.080ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:43177 req_id = 502f1522-3433-473b-a3b7-c4cf7e6b3099 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63912025-04-30T06:35:46.080ZINFOtest_metrics (dropshot): request completed latency_us = 650 local_addr = [::1]:56237 method = GET remote_addr = [::1]:43177 req_id = 502f1522-3433-473b-a3b7-c4cf7e6b3099 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
63922025-04-30T06:35:46.081ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "502f1522-3433-473b-a3b7-c4cf7e6b3099", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
63932025-04-30T06:35:46.089ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63942025-04-30T06:35:46.089ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
63952025-04-30T06:35:46.089ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
63962025-04-30T06:35:46.098ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
63972025-04-30T06:35:46.098ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
63982025-04-30T06:35:46.098ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
63992025-04-30T06:35:46.098ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64002025-04-30T06:35:46.099ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64012025-04-30T06:35:46.100ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64022025-04-30T06:35:46.110ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64032025-04-30T06:35:46.110ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64042025-04-30T06:35:46.110ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64052025-04-30T06:35:46.110ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64062025-04-30T06:35:46.113ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64072025-04-30T06:35:46.120ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64082025-04-30T06:35:46.120ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64092025-04-30T06:35:46.120ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64102025-04-30T06:35:46.122ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64112025-04-30T06:35:46.122ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64122025-04-30T06:35:46.122ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64132025-04-30T06:35:46.123ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64142025-04-30T06:35:46.147ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
64152025-04-30T06:35:46.147ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
64162025-04-30T06:35:46.147ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 431.468623ms iteration = 17 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
64172025-04-30T06:35:46.147ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 18 reason = Signaled
64182025-04-30T06:35:46.147ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
64192025-04-30T06:35:46.147ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64202025-04-30T06:35:46.147ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64212025-04-30T06:35:46.147ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64222025-04-30T06:35:46.148ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64232025-04-30T06:35:46.151ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
64242025-04-30T06:35:46.151ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64252025-04-30T06:35:46.156ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
64262025-04-30T06:35:46.156ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
64272025-04-30T06:35:46.156ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
64282025-04-30T06:35:46.157ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
64292025-04-30T06:35:46.161ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
64302025-04-30T06:35:46.161ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
64312025-04-30T06:35:46.161ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 34 35 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 45..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53989 req_id = 53b74804-95b6-4fcc-8385-8e72b99c028d
64322025-04-30T06:35:46.162ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13365,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53989 req_id = 53b74804-95b6-4fcc-8385-8e72b99c028d
64332025-04-30T06:35:46.162ZDEBGtest_metrics (store): query key key = _mgs._tcp
64342025-04-30T06:35:46.162ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64352025-04-30T06:35:46.162ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53989 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 53b74804-95b6-4fcc-8385-8e72b99c028d
64362025-04-30T06:35:46.163ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3810, tv_nsec: 962080949 } })
64372025-04-30T06:35:46.163ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
64382025-04-30T06:35:46.163ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b9 cb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:50024 req_id = 6a7dfc27-2b6c-4c73-80f9-bfd6a8191d03
64392025-04-30T06:35:46.163ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47563,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50024 req_id = 6a7dfc27-2b6c-4c73-80f9-bfd6a8191d03
64402025-04-30T06:35:46.163ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
64412025-04-30T06:35:46.163ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:50024 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6a7dfc27-2b6c-4c73-80f9-bfd6a8191d03
64422025-04-30T06:35:46.230ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
64432025-04-30T06:35:46.231ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
64442025-04-30T06:35:46.231ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
64452025-04-30T06:35:46.231ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64462025-04-30T06:35:46.231ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64472025-04-30T06:35:46.232ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64482025-04-30T06:35:46.232ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 9d 63 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .c..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58904 req_id = 0e0cb044-6484-4a25-91ae-4154ccdc3912
64492025-04-30T06:35:46.232ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 40291,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58904 req_id = 0e0cb044-6484-4a25-91ae-4154ccdc3912
64502025-04-30T06:35:46.232ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
64512025-04-30T06:35:46.233ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 40291, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58904 req_id = 0e0cb044-6484-4a25-91ae-4154ccdc3912
64522025-04-30T06:35:46.233ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
64532025-04-30T06:35:46.233ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
64542025-04-30T06:35:46.233ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
64552025-04-30T06:35:46.235ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
64562025-04-30T06:35:46.241ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
64572025-04-30T06:35:46.241ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64582025-04-30T06:35:46.241ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64592025-04-30T06:35:46.241ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64602025-04-30T06:35:46.245ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
64612025-04-30T06:35:46.245ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
64622025-04-30T06:35:46.247ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
64632025-04-30T06:35:46.247ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6bbb39f-73b7-4e6b-9295-6367f1d0489c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64642025-04-30T06:35:46.248ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
64652025-04-30T06:35:46.252ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd7d40ca-66d0-47e0-bfb9-66a91e8d4b2c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64662025-04-30T06:35:46.252ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
64672025-04-30T06:35:46.255ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdd23f9b-29cb-49dc-a690-aa24d0bc5149", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64682025-04-30T06:35:46.255ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
64692025-04-30T06:35:46.257ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ae632a5-f9fb-47d3-b9c8-3401bf609123", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64702025-04-30T06:35:46.257ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
64712025-04-30T06:35:46.260ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "689a8703-d1c4-43f1-8fdd-de2175ae08f5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64722025-04-30T06:35:46.260ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
64732025-04-30T06:35:46.264ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1db1a427-81d3-40dc-a7dd-dd5e0ba4c7f7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64742025-04-30T06:35:46.264ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
64752025-04-30T06:35:46.267ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d845c77-3d58-4e69-bac7-4360d67738d4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64762025-04-30T06:35:46.267ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
64772025-04-30T06:35:46.271ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "efb47468-8eee-49eb-939b-acee3030070c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64782025-04-30T06:35:46.271ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
64792025-04-30T06:35:46.273ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b8069e9-3113-4114-bf46-f4b8d3e8e283", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64802025-04-30T06:35:46.273ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
64812025-04-30T06:35:46.275ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d5ffc56-4712-459d-8918-0d7ec9405e1a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64822025-04-30T06:35:46.276ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
64832025-04-30T06:35:46.279ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e34fd07-bb81-44dc-a309-95df0e82622a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64842025-04-30T06:35:46.279ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
64852025-04-30T06:35:46.280ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
64862025-04-30T06:35:46.281ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cddcd3da-007d-40f1-8e29-e4f214700119", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64872025-04-30T06:35:46.282ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
64882025-04-30T06:35:46.283ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d99857e1-5739-42a8-a4d6-3b48727135d8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64892025-04-30T06:35:46.283ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
64902025-04-30T06:35:46.285ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33b02423-a9ac-45b0-890d-d015fbd5fe98", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64912025-04-30T06:35:46.285ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
64922025-04-30T06:35:46.287ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2f3f7a7-1014-48f8-b3ff-dc83931dd45e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64932025-04-30T06:35:46.287ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
64942025-04-30T06:35:46.289ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "620a1f7f-6b62-4499-a112-ec09705e0d47", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64952025-04-30T06:35:46.289ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
64962025-04-30T06:35:46.290ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "690ad621-c73f-4d2d-bce6-df4dd802ee22", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64972025-04-30T06:35:46.290ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
64982025-04-30T06:35:46.292ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2122220-3ac1-4330-8c9c-23ddc28c20aa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
64992025-04-30T06:35:46.292ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
65002025-04-30T06:35:46.295ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c12e6773-ba68-48e9-9729-ac678381c814", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65012025-04-30T06:35:46.295ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
65022025-04-30T06:35:46.296ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5caab542-153a-4b05-a182-86332c704ff8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65032025-04-30T06:35:46.296ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
65042025-04-30T06:35:46.297ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8e7a2e4-5437-4a77-b74c-f1a90206dd6b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65052025-04-30T06:35:46.297ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
65062025-04-30T06:35:46.298ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63d6c69c-fb76-4615-b6b4-16ebbd298317", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65072025-04-30T06:35:46.298ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
65082025-04-30T06:35:46.300ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e265de79-a1d1-422a-9309-15e07240ee69", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65092025-04-30T06:35:46.300ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
65102025-04-30T06:35:46.302ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2031f4e-11a4-43da-b522-85bcd6cca1aa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65112025-04-30T06:35:46.302ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
65122025-04-30T06:35:46.304ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6ebd11a-dbc6-48eb-a6dd-69c5548a97a9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65132025-04-30T06:35:46.304ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
65142025-04-30T06:35:46.307ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55955671-dc12-4ce8-9181-b7b0b3d6390b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65152025-04-30T06:35:46.307ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
65162025-04-30T06:35:46.309ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e174e91-97c0-4c0b-9925-ec4a38d2f21b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65172025-04-30T06:35:46.309ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
65182025-04-30T06:35:46.312ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b521b07-9ac5-47ae-8327-66920311c0e8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65192025-04-30T06:35:46.312ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
65202025-04-30T06:35:46.314ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "51e39d09-0a62-4413-b21d-79afe7f8b2dc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65212025-04-30T06:35:46.314ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
65222025-04-30T06:35:46.318ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5699e805-1aeb-4e68-8381-d098793aae7d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65232025-04-30T06:35:46.318ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
65242025-04-30T06:35:46.319ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b00ef98-010b-45a5-8daa-0e91b317b407", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65252025-04-30T06:35:46.319ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
65262025-04-30T06:35:46.321ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eacc7c38-fa71-4824-8e2f-b8d659a2f3e1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65272025-04-30T06:35:46.321ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
65282025-04-30T06:35:46.322ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30fc5eac-e161-4b30-8638-e928dd7d93f9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65292025-04-30T06:35:46.322ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
65302025-04-30T06:35:46.323ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "112f47fe-85d7-4b1f-9ed1-154a5da05ede", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65312025-04-30T06:35:46.323ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
65322025-04-30T06:35:46.325ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb7dd4de-09ea-453a-810e-ddf754d1a761", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65332025-04-30T06:35:46.325ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
65342025-04-30T06:35:46.327ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e179a0df-0ac7-40f3-a069-5805597d1255", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65352025-04-30T06:35:46.333ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
65362025-04-30T06:35:46.334ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65372025-04-30T06:35:46.334ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65382025-04-30T06:35:46.334ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65392025-04-30T06:35:46.334ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65402025-04-30T06:35:46.335ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65412025-04-30T06:35:46.338ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65422025-04-30T06:35:46.346ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
65432025-04-30T06:35:46.346ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65442025-04-30T06:35:46.346ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65452025-04-30T06:35:46.347ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65462025-04-30T06:35:46.353ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65472025-04-30T06:35:46.353ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65482025-04-30T06:35:46.354ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65492025-04-30T06:35:46.378ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b4c47d7-ea58-44f5-901f-60c8832cd9ad", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65502025-04-30T06:35:46.378ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
65512025-04-30T06:35:46.398ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc7ca4a0-cd89-4a8d-bfd2-c41328177e51", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65522025-04-30T06:35:46.398ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
65532025-04-30T06:35:46.400ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ff55d64-8d27-45eb-bfb4-37a3c5955d47", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65542025-04-30T06:35:46.400ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
65552025-04-30T06:35:46.404ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "320f5042-1563-499f-9db5-56026b158d82", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65562025-04-30T06:35:46.405ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
65572025-04-30T06:35:46.406ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
65582025-04-30T06:35:46.408ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9ce9851-2bb3-4b9f-b82e-d208cd6cd96f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65592025-04-30T06:35:46.408ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
65602025-04-30T06:35:46.409ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43d7659f-f625-4a8b-b3e9-8c3d22c3853b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65612025-04-30T06:35:46.410ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
65622025-04-30T06:35:46.411ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55d18e1f-c6fc-48c0-a154-adaaf9a58fa0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65632025-04-30T06:35:46.411ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
65642025-04-30T06:35:46.412ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "38d3fe19-dec8-4d61-b82c-437bab4acaea", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65652025-04-30T06:35:46.412ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
65662025-04-30T06:35:46.413ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9ed8d2a-41a9-4eea-ac30-c986f8ad323a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65672025-04-30T06:35:46.413ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65682025-04-30T06:35:46.413ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65692025-04-30T06:35:46.413ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65702025-04-30T06:35:46.413ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65712025-04-30T06:35:46.415ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65722025-04-30T06:35:46.417ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65732025-04-30T06:35:46.424ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65742025-04-30T06:35:46.424ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
65752025-04-30T06:35:46.425ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65762025-04-30T06:35:46.425ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65772025-04-30T06:35:46.425ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65782025-04-30T06:35:46.428ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65792025-04-30T06:35:46.433ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
65802025-04-30T06:35:46.439ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
65812025-04-30T06:35:46.439ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
65822025-04-30T06:35:46.439ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
65832025-04-30T06:35:46.440ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
65842025-04-30T06:35:46.497ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
65852025-04-30T06:35:46.497ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
65862025-04-30T06:35:46.497ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
65872025-04-30T06:35:46.497ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
65882025-04-30T06:35:46.497ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
65892025-04-30T06:35:46.498ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
65902025-04-30T06:35:46.498ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
65912025-04-30T06:35:46.498ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:34847 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
65922025-04-30T06:35:46.499ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:34847 req_id = a108c92c-7333-40be-ba24-c3593251d5d7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65932025-04-30T06:35:46.500ZINFOtest_metrics (dropshot): request completed latency_us = 605 local_addr = [::1]:56237 method = GET remote_addr = [::1]:34847 req_id = a108c92c-7333-40be-ba24-c3593251d5d7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
65942025-04-30T06:35:46.500ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a108c92c-7333-40be-ba24-c3593251d5d7", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:45 GMT"} })
65952025-04-30T06:35:46.503ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65962025-04-30T06:35:46.503ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
65972025-04-30T06:35:46.503ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
65982025-04-30T06:35:46.507ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
65992025-04-30T06:35:46.507ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66002025-04-30T06:35:46.507ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66012025-04-30T06:35:46.507ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66022025-04-30T06:35:46.508ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66032025-04-30T06:35:46.508ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66042025-04-30T06:35:46.526ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66052025-04-30T06:35:46.526ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66062025-04-30T06:35:46.526ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66072025-04-30T06:35:46.527ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66082025-04-30T06:35:46.528ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66092025-04-30T06:35:46.528ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66102025-04-30T06:35:46.530ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66112025-04-30T06:35:46.534ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66122025-04-30T06:35:46.534ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66132025-04-30T06:35:46.534ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66142025-04-30T06:35:46.534ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66152025-04-30T06:35:46.535ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66162025-04-30T06:35:46.536ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66172025-04-30T06:35:46.536ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66182025-04-30T06:35:46.536ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
66192025-04-30T06:35:46.537ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66202025-04-30T06:35:46.537ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66212025-04-30T06:35:46.537ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66222025-04-30T06:35:46.564ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
66232025-04-30T06:35:46.565ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
66242025-04-30T06:35:46.565ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 417.776382ms iteration = 18 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
66252025-04-30T06:35:46.565ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 19 reason = Signaled
66262025-04-30T06:35:46.565ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
66272025-04-30T06:35:46.565ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66282025-04-30T06:35:46.565ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66292025-04-30T06:35:46.565ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66302025-04-30T06:35:46.566ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66312025-04-30T06:35:46.568ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66322025-04-30T06:35:46.568ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66332025-04-30T06:35:46.569ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
66342025-04-30T06:35:46.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
66352025-04-30T06:35:46.570ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66362025-04-30T06:35:46.575ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
66372025-04-30T06:35:46.575ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
66382025-04-30T06:35:46.575ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
66392025-04-30T06:35:46.575ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
66402025-04-30T06:35:46.576ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
66412025-04-30T06:35:46.579ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
66422025-04-30T06:35:46.579ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
66432025-04-30T06:35:46.580ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
66442025-04-30T06:35:46.580ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
66452025-04-30T06:35:46.582ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 83 64 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .d..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52641 req_id = f4eba6c4-752c-44b1-b07e-ae30bd3f5128
66462025-04-30T06:35:46.582ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 33636,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52641 req_id = f4eba6c4-752c-44b1-b07e-ae30bd3f5128
66472025-04-30T06:35:46.582ZDEBGtest_metrics (store): query key key = _mgs._tcp
66482025-04-30T06:35:46.583ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66492025-04-30T06:35:46.584ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52641 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f4eba6c4-752c-44b1-b07e-ae30bd3f5128
66502025-04-30T06:35:46.586ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
66512025-04-30T06:35:46.586ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
66522025-04-30T06:35:46.587ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3811, tv_nsec: 386085641 } })
66532025-04-30T06:35:46.587ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
66542025-04-30T06:35:46.588ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b6 e8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59590 req_id = fac29314-725b-4067-9a04-325d1ae7fad5
66552025-04-30T06:35:46.588ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46824,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59590 req_id = fac29314-725b-4067-9a04-325d1ae7fad5
66562025-04-30T06:35:46.589ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
66572025-04-30T06:35:46.589ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59590 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fac29314-725b-4067-9a04-325d1ae7fad5
66582025-04-30T06:35:46.633ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
66592025-04-30T06:35:46.633ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
66602025-04-30T06:35:46.633ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
66612025-04-30T06:35:46.633ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66622025-04-30T06:35:46.633ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66632025-04-30T06:35:46.634ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66642025-04-30T06:35:46.634ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 01 5a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .Z..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60949 req_id = aee012ff-1e3d-4794-a9e2-d1e04a4064d5
66652025-04-30T06:35:46.634ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 346,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60949 req_id = aee012ff-1e3d-4794-a9e2-d1e04a4064d5
66662025-04-30T06:35:46.635ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
66672025-04-30T06:35:46.635ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 346, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60949 req_id = aee012ff-1e3d-4794-a9e2-d1e04a4064d5
66682025-04-30T06:35:46.635ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
66692025-04-30T06:35:46.635ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
66702025-04-30T06:35:46.635ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
66712025-04-30T06:35:46.637ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
66722025-04-30T06:35:46.643ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
66732025-04-30T06:35:46.643ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66742025-04-30T06:35:46.643ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66752025-04-30T06:35:46.645ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66762025-04-30T06:35:46.645ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbdefe13-75b5-4af7-875f-3535262ffa3b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66772025-04-30T06:35:46.645ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
66782025-04-30T06:35:46.647ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b16ff1ba-5b42-4caf-941c-e35e64c9ed9c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66792025-04-30T06:35:46.647ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
66802025-04-30T06:35:46.648ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
66812025-04-30T06:35:46.648ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
66822025-04-30T06:35:46.648ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
66832025-04-30T06:35:46.650ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e72f4c1b-38ab-479a-a4e0-c3dc6abddb42", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66842025-04-30T06:35:46.650ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
66852025-04-30T06:35:46.653ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0b1404b9-7d56-4e81-8915-aff4f2c80f5e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66862025-04-30T06:35:46.653ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
66872025-04-30T06:35:46.655ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab3af48f-deda-4c6c-af48-a58cd8b80566", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66882025-04-30T06:35:46.655ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
66892025-04-30T06:35:46.657ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "88a91750-0de4-4e20-a96e-f7a99d3cf32a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66902025-04-30T06:35:46.657ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
66912025-04-30T06:35:46.662ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f5907b00-7fd0-4434-a436-a58b402f992f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66922025-04-30T06:35:46.662ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
66932025-04-30T06:35:46.669ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a44bd8c6-93b0-4a2a-8037-d881865ef080", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66942025-04-30T06:35:46.669ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
66952025-04-30T06:35:46.669ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
66962025-04-30T06:35:46.671ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e279923-a81f-4d3d-a6e2-826913947fd7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66972025-04-30T06:35:46.672ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
66982025-04-30T06:35:46.674ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "019ae229-4537-4ca9-92ca-2e6ea9daeb3e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
66992025-04-30T06:35:46.674ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
67002025-04-30T06:35:46.675ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "10d65157-c9c7-4e97-961a-c437d463cc4a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67012025-04-30T06:35:46.675ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
67022025-04-30T06:35:46.679ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bba4381f-5aa7-4b03-afbe-29670d5b95d7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67032025-04-30T06:35:46.679ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
67042025-04-30T06:35:46.680ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
67052025-04-30T06:35:46.681ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a678f914-ad01-4f92-bfe0-8dabbc10a06e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67062025-04-30T06:35:46.681ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
67072025-04-30T06:35:46.684ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16e1978f-664e-462e-8300-ef6de0a7af6e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67082025-04-30T06:35:46.684ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
67092025-04-30T06:35:46.686ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b20d22b7-a1a7-4b83-a147-841f70d21202", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67102025-04-30T06:35:46.686ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
67112025-04-30T06:35:46.688ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f0227fe-9143-47b1-a3ba-0103ba026e5f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67122025-04-30T06:35:46.688ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
67132025-04-30T06:35:46.690ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fed2d11b-0135-46e7-8ee4-14d492542537", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67142025-04-30T06:35:46.690ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
67152025-04-30T06:35:46.692ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2503386-2ca9-47bb-be2c-0c699dfb92c3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67162025-04-30T06:35:46.692ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
67172025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c898fed-a7e2-4f06-afe1-1158b55d8d0b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67182025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
67192025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2cd2df87-5b0b-498b-b315-62a8a9071ce0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67202025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
67212025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e63cf5fb-30d9-4ffb-b53f-1563529b1852", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67222025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
67232025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf44c2a0-d5c3-4095-b7b8-1625f8f5b7ba", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67242025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
67252025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92a3198a-8e99-49cf-8d92-ecbeabd18753", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67262025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
67272025-04-30T06:35:46.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aed70fda-b5b4-4854-bf8f-ea6a1c7522aa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67282025-04-30T06:35:46.701ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
67292025-04-30T06:35:46.703ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "789338bd-2b1d-40f3-ab78-d8c2f2f4848f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67302025-04-30T06:35:46.703ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
67312025-04-30T06:35:46.707ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "edd26864-7927-4cc1-b265-df55fff6b644", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67322025-04-30T06:35:46.707ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
67332025-04-30T06:35:46.709ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4350763f-7b87-498f-9b70-3720aa6c9c76", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67342025-04-30T06:35:46.709ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
67352025-04-30T06:35:46.711ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7d6931b-cd5d-4851-b616-621ce7a2871a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67362025-04-30T06:35:46.711ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
67372025-04-30T06:35:46.713ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5a554ef-5e09-4926-a7b0-3d0b1215b2a1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67382025-04-30T06:35:46.713ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
67392025-04-30T06:35:46.715ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "879de347-0cc6-4a73-8827-93f394d3901b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67402025-04-30T06:35:46.715ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
67412025-04-30T06:35:46.716ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e78b0f0b-9d21-413d-9d91-167d02dda252", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67422025-04-30T06:35:46.717ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
67432025-04-30T06:35:46.718ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f6b98ad-1161-4a17-acca-f1b42016ce61", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67442025-04-30T06:35:46.719ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
67452025-04-30T06:35:46.720ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "63a0d003-056e-4b97-8b15-171f870f7808", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67462025-04-30T06:35:46.720ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
67472025-04-30T06:35:46.721ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cb09c0b-15bf-4f40-8bf0-198bd28ac4b9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67482025-04-30T06:35:46.721ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
67492025-04-30T06:35:46.723ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3c5511d-b39e-4e73-ae72-dd189f07c07f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67502025-04-30T06:35:46.723ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
67512025-04-30T06:35:46.726ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c38cc28-2f37-4e96-b5fd-40527c6c3b6f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67522025-04-30T06:35:46.727ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
67532025-04-30T06:35:46.728ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8e08e79-85cb-44c5-8eb2-9c6487ab82f2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67542025-04-30T06:35:46.728ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
67552025-04-30T06:35:46.731ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4e46058-8cc7-4fc2-9bd7-66a216457bd8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67562025-04-30T06:35:46.731ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
67572025-04-30T06:35:46.731ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
67582025-04-30T06:35:46.731ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67592025-04-30T06:35:46.731ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67602025-04-30T06:35:46.731ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
67612025-04-30T06:35:46.732ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67622025-04-30T06:35:46.733ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1069c51-6d9c-4cab-8442-cca306e0fc94", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67632025-04-30T06:35:46.734ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
67642025-04-30T06:35:46.734ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67652025-04-30T06:35:46.742ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
67662025-04-30T06:35:46.742ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67672025-04-30T06:35:46.742ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
67682025-04-30T06:35:46.743ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67692025-04-30T06:35:46.746ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
67702025-04-30T06:35:46.746ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
67712025-04-30T06:35:46.748ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
67722025-04-30T06:35:46.748ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c974a18d-70f6-46af-88f4-b9e61ad0f7ab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67732025-04-30T06:35:46.749ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
67742025-04-30T06:35:46.754ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2706c73c-d48c-4a9c-abd9-efa283f5ad06", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67752025-04-30T06:35:46.754ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
67762025-04-30T06:35:46.756ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1647fd7-3d8c-4236-9867-3bf139a9dde0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67772025-04-30T06:35:46.757ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
67782025-04-30T06:35:46.760ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1b44dd8-8a17-4f4d-868f-bad1f91b3029", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67792025-04-30T06:35:46.760ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
67802025-04-30T06:35:46.761ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c72330ba-b777-4914-b38b-4cbb2b811fca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67812025-04-30T06:35:46.761ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
67822025-04-30T06:35:46.762ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f30c9a13-9052-42fe-81dc-ef27058e4c5e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
67832025-04-30T06:35:46.763ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67842025-04-30T06:35:46.763ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67852025-04-30T06:35:46.763ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67862025-04-30T06:35:46.763ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67872025-04-30T06:35:46.764ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67882025-04-30T06:35:46.768ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67892025-04-30T06:35:46.773ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67902025-04-30T06:35:46.773ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
67912025-04-30T06:35:46.773ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
67922025-04-30T06:35:46.773ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67932025-04-30T06:35:46.773ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67942025-04-30T06:35:46.774ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
67952025-04-30T06:35:46.782ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
67962025-04-30T06:35:46.786ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
67972025-04-30T06:35:46.786ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
67982025-04-30T06:35:46.786ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
67992025-04-30T06:35:46.786ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68002025-04-30T06:35:46.856ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
68012025-04-30T06:35:46.856ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
68022025-04-30T06:35:46.857ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:32975 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
68032025-04-30T06:35:46.858ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:32975 req_id = 44bd75e7-98d7-4c8a-9740-eb0308efa9de sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68042025-04-30T06:35:46.858ZINFOtest_metrics (dropshot): request completed latency_us = 596 local_addr = [::1]:56237 method = GET remote_addr = [::1]:32975 req_id = 44bd75e7-98d7-4c8a-9740-eb0308efa9de response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
68052025-04-30T06:35:46.859ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "44bd75e7-98d7-4c8a-9740-eb0308efa9de", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68062025-04-30T06:35:46.862ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68072025-04-30T06:35:46.863ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
68082025-04-30T06:35:46.864ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
68092025-04-30T06:35:46.870ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68102025-04-30T06:35:46.870ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68112025-04-30T06:35:46.871ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68122025-04-30T06:35:46.871ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68132025-04-30T06:35:46.871ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68142025-04-30T06:35:46.878ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
68152025-04-30T06:35:46.879ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68162025-04-30T06:35:46.883ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68172025-04-30T06:35:46.883ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68182025-04-30T06:35:46.883ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68192025-04-30T06:35:46.883ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68202025-04-30T06:35:46.929ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
68212025-04-30T06:35:46.929ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68222025-04-30T06:35:46.929ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68232025-04-30T06:35:46.929ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
68242025-04-30T06:35:46.930ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68252025-04-30T06:35:46.935ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68262025-04-30T06:35:46.942ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
68272025-04-30T06:35:46.942ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68282025-04-30T06:35:46.942ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
68292025-04-30T06:35:46.943ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68302025-04-30T06:35:46.950ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
68312025-04-30T06:35:46.950ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
68322025-04-30T06:35:46.950ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 385.283001ms iteration = 19 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
68332025-04-30T06:35:46.950ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 20 reason = Signaled
68342025-04-30T06:35:46.950ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
68352025-04-30T06:35:46.950ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
68362025-04-30T06:35:46.950ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68372025-04-30T06:35:46.951ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68382025-04-30T06:35:46.951ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68392025-04-30T06:35:46.953ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
68402025-04-30T06:35:46.953ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
68412025-04-30T06:35:46.955ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
68422025-04-30T06:35:46.955ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
68432025-04-30T06:35:46.959ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
68442025-04-30T06:35:46.959ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
68452025-04-30T06:35:46.959ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
68462025-04-30T06:35:46.959ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
68472025-04-30T06:35:46.964ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
68482025-04-30T06:35:46.964ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
68492025-04-30T06:35:46.966ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: cf 58 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .X..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50213 req_id = ccac25dd-7f6f-475b-a703-6462b7fb0046
68502025-04-30T06:35:46.966ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 53080,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50213 req_id = ccac25dd-7f6f-475b-a703-6462b7fb0046
68512025-04-30T06:35:46.966ZDEBGtest_metrics (store): query key key = _mgs._tcp
68522025-04-30T06:35:46.966ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68532025-04-30T06:35:46.966ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50213 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ccac25dd-7f6f-475b-a703-6462b7fb0046
68542025-04-30T06:35:46.967ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3811, tv_nsec: 766206417 } })
68552025-04-30T06:35:46.967ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
68562025-04-30T06:35:46.967ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 14 a3 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55063 req_id = 2900cb5a-fbf6-4346-a891-1b8c2a86dad1
68572025-04-30T06:35:46.967ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5283,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55063 req_id = 2900cb5a-fbf6-4346-a891-1b8c2a86dad1
68582025-04-30T06:35:46.967ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
68592025-04-30T06:35:46.967ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55063 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2900cb5a-fbf6-4346-a891-1b8c2a86dad1
68602025-04-30T06:35:47.020ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
68612025-04-30T06:35:47.021ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2b 26 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c +&..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51164 req_id = 3d7de410-3b36-41ad-903d-31844130eef5
68622025-04-30T06:35:47.021ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11046,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51164 req_id = 3d7de410-3b36-41ad-903d-31844130eef5
68632025-04-30T06:35:47.021ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
68642025-04-30T06:35:47.022ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11046, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51164 req_id = 3d7de410-3b36-41ad-903d-31844130eef5
68652025-04-30T06:35:47.022ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
68662025-04-30T06:35:47.022ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
68672025-04-30T06:35:47.022ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
68682025-04-30T06:35:47.026ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "1466d20f-f460-49f0-8934-66f9152f0004", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68692025-04-30T06:35:47.027ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
68702025-04-30T06:35:47.029ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5408c0ef-80c8-4f47-9782-5a37566bccd0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68712025-04-30T06:35:47.029ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
68722025-04-30T06:35:47.033ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "739e149f-06c9-4a86-88c1-ff404ca7c559", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68732025-04-30T06:35:47.033ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
68742025-04-30T06:35:47.037ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "db588464-8df9-4c0d-b1ef-f8c96364e3d6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68752025-04-30T06:35:47.037ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
68762025-04-30T06:35:47.040ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e160442-8a0f-4199-a510-b2c49ab35ebe", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68772025-04-30T06:35:47.040ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
68782025-04-30T06:35:47.042ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cc32873-887b-4adf-97be-0c3206c9d4f2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68792025-04-30T06:35:47.042ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
68802025-04-30T06:35:47.051ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
68812025-04-30T06:35:47.052ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f364183-99cb-430f-82f3-d105a67a8f61", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68822025-04-30T06:35:47.052ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
68832025-04-30T06:35:47.058ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "269d6ea3-bf6c-4fb0-b40b-191d85027db1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68842025-04-30T06:35:47.058ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
68852025-04-30T06:35:47.060ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1884835e-d984-419d-b509-ae076a0e66b6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68862025-04-30T06:35:47.060ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
68872025-04-30T06:35:47.062ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ae81fd2-a247-41bd-bd20-0be1b9160f8d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68882025-04-30T06:35:47.062ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
68892025-04-30T06:35:47.065ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fd082e7-76b4-482d-bbd5-1104bf792b4b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68902025-04-30T06:35:47.066ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
68912025-04-30T06:35:47.067ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b60a445-9762-4efc-8adb-c8625b573ea8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68922025-04-30T06:35:47.068ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
68932025-04-30T06:35:47.075ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e111d3ea-51ab-4d0e-a39e-48982d151b15", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68942025-04-30T06:35:47.076ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
68952025-04-30T06:35:47.089ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76fd4fdc-cf9d-4dbf-9aca-168a7d0e6824", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68962025-04-30T06:35:47.090ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
68972025-04-30T06:35:47.096ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8553c74-fa79-4e3d-8d9e-70ddb00a7d7a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
68982025-04-30T06:35:47.097ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
68992025-04-30T06:35:47.102ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69002025-04-30T06:35:47.102ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69012025-04-30T06:35:47.102ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69022025-04-30T06:35:47.102ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69032025-04-30T06:35:47.103ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69042025-04-30T06:35:47.103ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7af36203-b317-4daf-90f2-cf11b14f85aa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69052025-04-30T06:35:47.104ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
69062025-04-30T06:35:47.108ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6549a16b-6cd3-4d2f-a75c-bd3e354f2be6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69072025-04-30T06:35:47.108ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
69082025-04-30T06:35:47.110ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59ceb5d2-6788-440c-b09c-0c3896ffc520", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69092025-04-30T06:35:47.111ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
69102025-04-30T06:35:47.111ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69112025-04-30T06:35:47.118ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69122025-04-30T06:35:47.118ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69132025-04-30T06:35:47.118ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69142025-04-30T06:35:47.119ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69152025-04-30T06:35:47.123ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18ecdbc0-921f-4689-a545-d320d9accd92", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69162025-04-30T06:35:47.123ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
69172025-04-30T06:35:47.123ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69182025-04-30T06:35:47.124ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69192025-04-30T06:35:47.124ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69202025-04-30T06:35:47.126ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8b98e11-a3f9-42e9-b063-b4f8a1732ff1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69212025-04-30T06:35:47.126ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
69222025-04-30T06:35:47.129ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6caa07db-fc26-4a1b-8576-0712a6355197", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69232025-04-30T06:35:47.129ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
69242025-04-30T06:35:47.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5349263b-71a2-47f7-8f0d-f63cbd29ed8c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69252025-04-30T06:35:47.133ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
69262025-04-30T06:35:47.134ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1660be23-9018-47f5-b9f7-3d19819b4fdb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69272025-04-30T06:35:47.135ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
69282025-04-30T06:35:47.136ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d03347f0-c7ad-4a5f-9ade-7dd43f8a1140", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69292025-04-30T06:35:47.137ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
69302025-04-30T06:35:47.139ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de885dfb-a0ae-4174-ba04-9233e24a6aad", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69312025-04-30T06:35:47.139ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
69322025-04-30T06:35:47.141ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "19d20ac1-6493-457f-89b6-43b76d8e3a94", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69332025-04-30T06:35:47.142ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
69342025-04-30T06:35:47.145ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a4e1db5-6a34-46fe-84f4-96743bda85b6", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69352025-04-30T06:35:47.145ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
69362025-04-30T06:35:47.149ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e44ea082-0b21-4c13-9ead-22fdd1b67478", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69372025-04-30T06:35:47.149ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
69382025-04-30T06:35:47.153ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0a31cc5-af42-4a82-823c-ad7a9463e0ae", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69392025-04-30T06:35:47.153ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
69402025-04-30T06:35:47.158ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
69412025-04-30T06:35:47.161ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4eecbc8-84b9-48c8-a1dc-61666871ebc3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69422025-04-30T06:35:47.161ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
69432025-04-30T06:35:47.163ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ea9ecec-2dec-465e-830b-8c025414d931", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69442025-04-30T06:35:47.163ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
69452025-04-30T06:35:47.173ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a1f3c96-3353-4ef1-9baa-6c8bb5381f96", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69462025-04-30T06:35:47.173ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
69472025-04-30T06:35:47.175ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7c23b79-615a-4076-9f73-b05715f4b7e2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69482025-04-30T06:35:47.177ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
69492025-04-30T06:35:47.178ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e341e484-0bb4-479e-964f-f9665b17b9e5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69502025-04-30T06:35:47.178ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
69512025-04-30T06:35:47.179ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bac5315-e123-44fb-addc-324f3cc17cde", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69522025-04-30T06:35:47.180ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
69532025-04-30T06:35:47.182ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6789b64-b417-4cf6-b4e7-ba6edef16470", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69542025-04-30T06:35:47.182ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
69552025-04-30T06:35:47.186ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0962ebe6-e98e-4961-891e-ee896e997383", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69562025-04-30T06:35:47.186ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
69572025-04-30T06:35:47.188ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "647d6b62-8f04-413f-9365-223bb159867a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69582025-04-30T06:35:47.188ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
69592025-04-30T06:35:47.192ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7106556-d200-49cd-b932-57f99d9227e7", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69602025-04-30T06:35:47.192ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
69612025-04-30T06:35:47.197ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e596481a-f0c0-48d8-ad73-ecbba4714826", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69622025-04-30T06:35:47.198ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
69632025-04-30T06:35:47.201ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12a77a81-5d59-4fb1-af6f-5d10986185ce", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69642025-04-30T06:35:47.201ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
69652025-04-30T06:35:47.202ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "132997b6-54e9-4952-821c-549410ccdc1e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69662025-04-30T06:35:47.202ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
69672025-04-30T06:35:47.203ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "beeb39bb-1f34-4d74-a5c3-9baf906f5afd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69682025-04-30T06:35:47.203ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
69692025-04-30T06:35:47.207ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a9c9bfd-61ce-4667-b589-fc576b1dfaab", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69702025-04-30T06:35:47.207ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
69712025-04-30T06:35:47.209ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ba3fc9f-030a-4623-95cb-f00b21d9bca5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
69722025-04-30T06:35:47.209ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69732025-04-30T06:35:47.209ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69742025-04-30T06:35:47.209ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69752025-04-30T06:35:47.209ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69762025-04-30T06:35:47.210ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69772025-04-30T06:35:47.211ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
69782025-04-30T06:35:47.211ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69792025-04-30T06:35:47.211ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69802025-04-30T06:35:47.211ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69812025-04-30T06:35:47.213ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69822025-04-30T06:35:47.215ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69832025-04-30T06:35:47.221ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69842025-04-30T06:35:47.221ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
69852025-04-30T06:35:47.221ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
69862025-04-30T06:35:47.221ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69872025-04-30T06:35:47.221ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69882025-04-30T06:35:47.222ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69892025-04-30T06:35:47.222ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69902025-04-30T06:35:47.232ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
69912025-04-30T06:35:47.232ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
69922025-04-30T06:35:47.232ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
69932025-04-30T06:35:47.232ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
69942025-04-30T06:35:47.234ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
69952025-04-30T06:35:47.239ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
69962025-04-30T06:35:47.239ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
69972025-04-30T06:35:47.239ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
69982025-04-30T06:35:47.240ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
69992025-04-30T06:35:47.241ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70002025-04-30T06:35:47.241ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70012025-04-30T06:35:47.242ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70022025-04-30T06:35:47.295ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
70032025-04-30T06:35:47.296ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
70042025-04-30T06:35:47.296ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:57656 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
70052025-04-30T06:35:47.296ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
70062025-04-30T06:35:47.297ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70072025-04-30T06:35:47.299ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ae28b038-0768-48e7-a604-2ce34bb281b7 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70082025-04-30T06:35:47.299ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:57656 req_id = 123c3324-d0c5-4cc9-8e2b-7198efd5baee sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70092025-04-30T06:35:47.300ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ae28b038-0768-48e7-a604-2ce34bb281b7 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70102025-04-30T06:35:47.300ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ae28b038-0768-48e7-a604-2ce34bb281b7 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70112025-04-30T06:35:47.301ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ae28b038-0768-48e7-a604-2ce34bb281b7 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70122025-04-30T06:35:47.302ZINFOtest_metrics (dropshot): request completed latency_us = 2306 local_addr = [::1]:56237 method = GET remote_addr = [::1]:57656 req_id = 123c3324-d0c5-4cc9-8e2b-7198efd5baee response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
70132025-04-30T06:35:47.303ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "123c3324-d0c5-4cc9-8e2b-7198efd5baee", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
70142025-04-30T06:35:47.307ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70152025-04-30T06:35:47.307ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
70162025-04-30T06:35:47.307ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
70172025-04-30T06:35:47.309ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70182025-04-30T06:35:47.309ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70192025-04-30T06:35:47.309ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70202025-04-30T06:35:47.310ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70212025-04-30T06:35:47.310ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70222025-04-30T06:35:47.313ZINFOtest_metrics (dropshot_internal): request completed latency_us = 14216 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ae28b038-0768-48e7-a604-2ce34bb281b7 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
70232025-04-30T06:35:47.314ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70242025-04-30T06:35:47.317ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70252025-04-30T06:35:47.317ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70262025-04-30T06:35:47.318ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70272025-04-30T06:35:47.319ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70282025-04-30T06:35:47.319ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae28b038-0768-48e7-a604-2ce34bb281b7", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
70292025-04-30T06:35:47.319ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70302025-04-30T06:35:47.320ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3bff54a3-8603-4c0d-abf7-a127c56b97e3 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70312025-04-30T06:35:47.321ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3bff54a3-8603-4c0d-abf7-a127c56b97e3 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70322025-04-30T06:35:47.321ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3bff54a3-8603-4c0d-abf7-a127c56b97e3 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70332025-04-30T06:35:47.321ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3bff54a3-8603-4c0d-abf7-a127c56b97e3 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70342025-04-30T06:35:47.325ZINFOtest_metrics (dropshot_internal): request completed latency_us = 5020 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3bff54a3-8603-4c0d-abf7-a127c56b97e3 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
70352025-04-30T06:35:47.326ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bff54a3-8603-4c0d-abf7-a127c56b97e3", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:46 GMT"} })
70362025-04-30T06:35:47.326ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
70372025-04-30T06:35:47.326ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70382025-04-30T06:35:47.326ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
70392025-04-30T06:35:47.358ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
70402025-04-30T06:35:47.414ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
70412025-04-30T06:35:47.414ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70422025-04-30T06:35:47.414ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70432025-04-30T06:35:47.414ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70442025-04-30T06:35:47.415ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70452025-04-30T06:35:47.423ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70462025-04-30T06:35:47.431ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
70472025-04-30T06:35:47.432ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70482025-04-30T06:35:47.432ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70492025-04-30T06:35:47.432ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70502025-04-30T06:35:47.456ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
70512025-04-30T06:35:47.456ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
70522025-04-30T06:35:47.457ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
70532025-04-30T06:35:47.457ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
70542025-04-30T06:35:47.458ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
70552025-04-30T06:35:47.459ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 508.181047ms iteration = 20 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
70562025-04-30T06:35:47.459ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 21 reason = Signaled
70572025-04-30T06:35:47.459ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
70582025-04-30T06:35:47.459ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
70592025-04-30T06:35:47.459ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70602025-04-30T06:35:47.459ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70612025-04-30T06:35:47.459ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70622025-04-30T06:35:47.463ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
70632025-04-30T06:35:47.467ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
70642025-04-30T06:35:47.467ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
70652025-04-30T06:35:47.467ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
70662025-04-30T06:35:47.468ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
70672025-04-30T06:35:47.473ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
70682025-04-30T06:35:47.473ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
70692025-04-30T06:35:47.477ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b1 12 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63258 req_id = deb0503f-ee37-4693-b660-2fa58d5fd9c1
70702025-04-30T06:35:47.477ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45330,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63258 req_id = deb0503f-ee37-4693-b660-2fa58d5fd9c1
70712025-04-30T06:35:47.477ZDEBGtest_metrics (store): query key key = _mgs._tcp
70722025-04-30T06:35:47.477ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70732025-04-30T06:35:47.477ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63258 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = deb0503f-ee37-4693-b660-2fa58d5fd9c1
70742025-04-30T06:35:47.478ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3812, tv_nsec: 277241351 } })
70752025-04-30T06:35:47.478ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
70762025-04-30T06:35:47.479ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2e 17 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53743 req_id = 8d4c4cb8-769f-437f-895b-95b19304a978
70772025-04-30T06:35:47.479ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11799,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53743 req_id = 8d4c4cb8-769f-437f-895b-95b19304a978
70782025-04-30T06:35:47.479ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
70792025-04-30T06:35:47.479ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53743 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8d4c4cb8-769f-437f-895b-95b19304a978
70802025-04-30T06:35:47.526ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
70812025-04-30T06:35:47.527ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70822025-04-30T06:35:47.527ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70832025-04-30T06:35:47.528ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 3b 94 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ;............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49504 req_id = e5741e14-7b69-489a-91eb-a86c9a068b9f
70842025-04-30T06:35:47.528ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15252,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49504 req_id = e5741e14-7b69-489a-91eb-a86c9a068b9f
70852025-04-30T06:35:47.528ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
70862025-04-30T06:35:47.528ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 15252, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49504 req_id = e5741e14-7b69-489a-91eb-a86c9a068b9f
70872025-04-30T06:35:47.530ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
70882025-04-30T06:35:47.530ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
70892025-04-30T06:35:47.530ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
70902025-04-30T06:35:47.530ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70912025-04-30T06:35:47.531ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70922025-04-30T06:35:47.531ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70932025-04-30T06:35:47.531ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
70942025-04-30T06:35:47.542ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d84d564f-14c0-49ef-a435-f5a6d5d8c7be", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
70952025-04-30T06:35:47.543ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
70962025-04-30T06:35:47.548ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "757da5cd-49c5-48bd-be55-f270697ebab6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
70972025-04-30T06:35:47.548ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
70982025-04-30T06:35:47.550ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f43268dc-edea-48bc-9d74-fbf0e1f36db0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
70992025-04-30T06:35:47.550ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
71002025-04-30T06:35:47.551ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
71012025-04-30T06:35:47.554ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0f55aa8-b338-4503-b0ab-67d01e7e6d20", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71022025-04-30T06:35:47.554ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
71032025-04-30T06:35:47.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "388c84a2-5cba-49ff-bb67-67b61dac5410", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71042025-04-30T06:35:47.556ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
71052025-04-30T06:35:47.564ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "124cf242-2081-43a5-9ab2-5b2b805de859", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71062025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
71072025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "41eb70c1-2504-45ae-8cdd-38c0752eee75", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71082025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
71092025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6644be01-45c8-4a54-a545-ae7100147e94", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71102025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
71112025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b979bfe0-3c7d-4f4a-b58d-d61f5729eb4e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71122025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
71132025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af0ecded-9570-428e-8852-38778da1fb20", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71142025-04-30T06:35:47.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
71152025-04-30T06:35:47.567ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "079e6819-bdd7-4d51-9c4b-af7a92336b1b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71162025-04-30T06:35:47.567ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
71172025-04-30T06:35:47.567ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "81d030ea-d65b-4d87-866c-7a23295a3921", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71182025-04-30T06:35:47.567ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
71192025-04-30T06:35:47.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71202025-04-30T06:35:47.568ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71212025-04-30T06:35:47.568ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
71222025-04-30T06:35:47.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
71232025-04-30T06:35:47.572ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b0434eb-f7c0-45d0-9402-d58955345d51", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71242025-04-30T06:35:47.572ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
71252025-04-30T06:35:47.573ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
71262025-04-30T06:35:47.573ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
71272025-04-30T06:35:47.576ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f66bc5-7a66-4d7e-a34d-0a7400f12487", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71282025-04-30T06:35:47.576ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
71292025-04-30T06:35:47.577ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
71302025-04-30T06:35:47.577ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
71312025-04-30T06:35:47.580ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31b264a2-ec48-427a-8969-3c8232595f95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71322025-04-30T06:35:47.580ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
71332025-04-30T06:35:47.582ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1096724-f046-4444-9fcd-afccca03e996", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71342025-04-30T06:35:47.582ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
71352025-04-30T06:35:47.585ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb03ca02-5103-4ff6-a192-bdf64469edc4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71362025-04-30T06:35:47.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
71372025-04-30T06:35:47.587ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b48f7ae5-e707-481f-a8f9-26f248233f9f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71382025-04-30T06:35:47.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
71392025-04-30T06:35:47.590ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50aa7966-b263-4466-9e1f-b94f5028b187", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71402025-04-30T06:35:47.591ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
71412025-04-30T06:35:47.592ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db450be-9c1d-4380-ba09-d7ca5c1ae137", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71422025-04-30T06:35:47.592ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
71432025-04-30T06:35:47.593ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d71eddc2-64bf-44e6-881b-9cbab0849374", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71442025-04-30T06:35:47.594ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
71452025-04-30T06:35:47.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9f98540-78a7-4394-8f5d-cdc4a5d89559", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71462025-04-30T06:35:47.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
71472025-04-30T06:35:47.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d32b26d-9fc0-455d-afc5-3ea6d4036887", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71482025-04-30T06:35:47.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
71492025-04-30T06:35:47.597ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4995fc84-0091-47f4-88fc-51a1eb4a4421", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71502025-04-30T06:35:47.598ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
71512025-04-30T06:35:47.599ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcf6cd4d-e932-4009-be1a-f94773a4c1c4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71522025-04-30T06:35:47.599ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
71532025-04-30T06:35:47.602ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
71542025-04-30T06:35:47.602ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
71552025-04-30T06:35:47.602ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71562025-04-30T06:35:47.602ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71572025-04-30T06:35:47.603ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71582025-04-30T06:35:47.603ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f62725a6-03b9-4f57-876c-a135bfb03a6b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71592025-04-30T06:35:47.603ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
71602025-04-30T06:35:47.606ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
71612025-04-30T06:35:47.612ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
71622025-04-30T06:35:47.612ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71632025-04-30T06:35:47.612ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71642025-04-30T06:35:47.612ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71652025-04-30T06:35:47.613ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fde692f6-be3c-4e7c-9449-1e500e8aa166", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71662025-04-30T06:35:47.614ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
71672025-04-30T06:35:47.615ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
71682025-04-30T06:35:47.615ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
71692025-04-30T06:35:47.616ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
71702025-04-30T06:35:47.618ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b271d12c-00b0-46a6-af4c-18f5ec857295", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71712025-04-30T06:35:47.618ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
71722025-04-30T06:35:47.620ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a1e75cc-e399-41b2-9aaf-c7aced00f6ab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71732025-04-30T06:35:47.620ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
71742025-04-30T06:35:47.623ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3df9821-1dd1-4384-9031-94ff7850b7cc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71752025-04-30T06:35:47.623ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
71762025-04-30T06:35:47.625ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "75c03b4a-23ea-4844-9533-87d8724f940e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71772025-04-30T06:35:47.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
71782025-04-30T06:35:47.627ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eddd04e6-0992-4ef7-a668-6c4a11bbcb10", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71792025-04-30T06:35:47.628ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
71802025-04-30T06:35:47.630ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54a1446-192e-4c26-bb35-1ec5fd5ef567", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71812025-04-30T06:35:47.630ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
71822025-04-30T06:35:47.634ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8825b4e-6954-4c00-aa87-97ed3b488d47", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71832025-04-30T06:35:47.634ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
71842025-04-30T06:35:47.634ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f22b1b4-a6f3-466d-b71d-c9bc5204c46b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71852025-04-30T06:35:47.634ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
71862025-04-30T06:35:47.636ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33282b4d-bc47-4c5b-befc-89b6158f0813", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71872025-04-30T06:35:47.636ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
71882025-04-30T06:35:47.643ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56f726db-fdf6-4058-9330-f12b31490624", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71892025-04-30T06:35:47.643ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
71902025-04-30T06:35:47.644ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
71912025-04-30T06:35:47.648ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ca64851-dcc2-46b8-bb2e-034c34758c25", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71922025-04-30T06:35:47.648ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
71932025-04-30T06:35:47.650ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25790097-25bb-41e8-9d9d-416810d9162d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71942025-04-30T06:35:47.650ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
71952025-04-30T06:35:47.652ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8afe47f7-27b4-4e99-8fc1-6146fadecb12", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71962025-04-30T06:35:47.652ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
71972025-04-30T06:35:47.654ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3e6666e-46ef-48af-98f6-14140570617a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
71982025-04-30T06:35:47.654ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
71992025-04-30T06:35:47.660ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b83a346-7b81-41c2-bc40-13ab68ab9b89", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
72002025-04-30T06:35:47.660ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
72012025-04-30T06:35:47.662ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e558ee7-f7e5-4a33-9e6f-43146e884083", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
72022025-04-30T06:35:47.662ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
72032025-04-30T06:35:47.664ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e66a81-5165-4017-8767-020c221226ed", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
72042025-04-30T06:35:47.664ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
72052025-04-30T06:35:47.665ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48850231-ee04-48a1-b483-94301d07a2e6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
72062025-04-30T06:35:47.666ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72072025-04-30T06:35:47.666ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72082025-04-30T06:35:47.666ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72092025-04-30T06:35:47.666ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72102025-04-30T06:35:47.668ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72112025-04-30T06:35:47.669ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
72122025-04-30T06:35:47.670ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72132025-04-30T06:35:47.676ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72142025-04-30T06:35:47.676ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
72152025-04-30T06:35:47.676ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72162025-04-30T06:35:47.676ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72172025-04-30T06:35:47.676ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72182025-04-30T06:35:47.677ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72192025-04-30T06:35:47.679ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72202025-04-30T06:35:47.684ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
72212025-04-30T06:35:47.684ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72222025-04-30T06:35:47.684ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72232025-04-30T06:35:47.684ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72242025-04-30T06:35:47.732ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
72252025-04-30T06:35:47.732ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
72262025-04-30T06:35:47.732ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72272025-04-30T06:35:47.732ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72282025-04-30T06:35:47.732ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72292025-04-30T06:35:47.732ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72302025-04-30T06:35:47.733ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72312025-04-30T06:35:47.733ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:48082 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
72322025-04-30T06:35:47.735ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:48082 req_id = f2b13245-960f-498c-b68b-def95cd5960e sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72332025-04-30T06:35:47.735ZINFOtest_metrics (dropshot): request completed latency_us = 541 local_addr = [::1]:56237 method = GET remote_addr = [::1]:48082 req_id = f2b13245-960f-498c-b68b-def95cd5960e response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
72342025-04-30T06:35:47.736ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2b13245-960f-498c-b68b-def95cd5960e", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
72352025-04-30T06:35:47.738ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72362025-04-30T06:35:47.739ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
72372025-04-30T06:35:47.739ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
72382025-04-30T06:35:47.741ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72392025-04-30T06:35:47.742ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72402025-04-30T06:35:47.742ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72412025-04-30T06:35:47.742ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72422025-04-30T06:35:47.742ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72432025-04-30T06:35:47.743ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72442025-04-30T06:35:47.748ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72452025-04-30T06:35:47.748ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72462025-04-30T06:35:47.748ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72472025-04-30T06:35:47.749ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72482025-04-30T06:35:47.752ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72492025-04-30T06:35:47.758ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72502025-04-30T06:35:47.758ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72512025-04-30T06:35:47.758ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72522025-04-30T06:35:47.759ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72532025-04-30T06:35:47.761ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72542025-04-30T06:35:47.761ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72552025-04-30T06:35:47.762ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72562025-04-30T06:35:47.785ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
72572025-04-30T06:35:47.786ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
72582025-04-30T06:35:47.786ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 327.324663ms iteration = 21 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
72592025-04-30T06:35:47.786ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 22 reason = Signaled
72602025-04-30T06:35:47.786ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
72612025-04-30T06:35:47.786ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72622025-04-30T06:35:47.786ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72632025-04-30T06:35:47.786ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72642025-04-30T06:35:47.787ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72652025-04-30T06:35:47.791ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72662025-04-30T06:35:47.794ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
72672025-04-30T06:35:47.795ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
72682025-04-30T06:35:47.795ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
72692025-04-30T06:35:47.795ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
72702025-04-30T06:35:47.805ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
72712025-04-30T06:35:47.805ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
72722025-04-30T06:35:47.807ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
72732025-04-30T06:35:47.809ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a0 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .J..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59613 req_id = 25dc3c7f-19bb-42a6-a9a0-db5a26ef7370
72742025-04-30T06:35:47.809ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41034,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59613 req_id = 25dc3c7f-19bb-42a6-a9a0-db5a26ef7370
72752025-04-30T06:35:47.811ZDEBGtest_metrics (store): query key key = _mgs._tcp
72762025-04-30T06:35:47.811ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72772025-04-30T06:35:47.811ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59613 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 25dc3c7f-19bb-42a6-a9a0-db5a26ef7370
72782025-04-30T06:35:47.812ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3812, tv_nsec: 611194931 } })
72792025-04-30T06:35:47.812ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
72802025-04-30T06:35:47.813ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 9a bb 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51848 req_id = 78effe5f-484d-4e0e-a460-ac88a3bfa0fa
72812025-04-30T06:35:47.813ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 39611,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51848 req_id = 78effe5f-484d-4e0e-a460-ac88a3bfa0fa
72822025-04-30T06:35:47.813ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
72832025-04-30T06:35:47.813ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51848 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 78effe5f-484d-4e0e-a460-ac88a3bfa0fa
72842025-04-30T06:35:47.882ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
72852025-04-30T06:35:47.882ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
72862025-04-30T06:35:47.882ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
72872025-04-30T06:35:47.882ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
72882025-04-30T06:35:47.882ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
72892025-04-30T06:35:47.883ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
72902025-04-30T06:35:47.884ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 19 30 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .0..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51948 req_id = a23d9a66-0444-42df-9aa4-e3d3ddfa2d0e
72912025-04-30T06:35:47.884ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6448,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51948 req_id = a23d9a66-0444-42df-9aa4-e3d3ddfa2d0e
72922025-04-30T06:35:47.884ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
72932025-04-30T06:35:47.884ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6448, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51948 req_id = a23d9a66-0444-42df-9aa4-e3d3ddfa2d0e
72942025-04-30T06:35:47.884ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
72952025-04-30T06:35:47.884ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
72962025-04-30T06:35:47.884ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
72972025-04-30T06:35:47.895ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
72982025-04-30T06:35:47.895ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
72992025-04-30T06:35:47.895ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73002025-04-30T06:35:47.895ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73012025-04-30T06:35:47.895ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73022025-04-30T06:35:47.895ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "af9bbe04-68af-4183-84ef-d97422dc9a7a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73032025-04-30T06:35:47.898ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
73042025-04-30T06:35:47.899ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73052025-04-30T06:35:47.899ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73062025-04-30T06:35:47.899ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73072025-04-30T06:35:47.901ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "749a0f70-4dbe-4e2f-8aea-63b0159f9982", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73082025-04-30T06:35:47.901ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
73092025-04-30T06:35:47.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65d61cb7-a0d9-47e1-8839-174b414b0507", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73102025-04-30T06:35:47.905ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
73112025-04-30T06:35:47.908ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d7a5463-8ade-425c-80f9-d4fd7df456e3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73122025-04-30T06:35:47.908ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
73132025-04-30T06:35:47.910ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "89b60c19-591d-4436-9772-97db860db965", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73142025-04-30T06:35:47.910ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
73152025-04-30T06:35:47.913ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40f1d082-b1a4-449c-a3f4-6f976fd7ac0d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73162025-04-30T06:35:47.914ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
73172025-04-30T06:35:47.915ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74a06f32-c4ff-4e8a-86b0-ff0bdfdc239e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73182025-04-30T06:35:47.915ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
73192025-04-30T06:35:47.917ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be1ab9ad-9af7-4472-bb36-f2c705b83ba3", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73202025-04-30T06:35:47.917ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
73212025-04-30T06:35:47.919ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16a34a3a-e961-4c96-91ca-368d940582df", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73222025-04-30T06:35:47.919ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
73232025-04-30T06:35:47.920ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4638ea5e-9338-4b89-a4ff-d02ea87ffbab", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73242025-04-30T06:35:47.920ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
73252025-04-30T06:35:47.921ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a26b3177-5eb1-4990-9276-0f2b2efbfe93", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73262025-04-30T06:35:47.921ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
73272025-04-30T06:35:47.923ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eeb59ba6-016f-45bb-aaa8-74b23d4005ac", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73282025-04-30T06:35:47.923ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
73292025-04-30T06:35:47.925ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c7c1760-8e2e-4716-8e28-4ed584953d81", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73302025-04-30T06:35:47.925ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
73312025-04-30T06:35:47.929ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "49ef838a-fff5-4ea4-bf5b-27b57fcd3c69", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73322025-04-30T06:35:47.929ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
73332025-04-30T06:35:47.929ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
73342025-04-30T06:35:47.932ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d754e59-fdaa-4edd-984e-582ee52ac577", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73352025-04-30T06:35:47.932ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
73362025-04-30T06:35:47.936ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9c6a18-ac59-4d82-b824-4037f8a2a08d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73372025-04-30T06:35:47.936ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
73382025-04-30T06:35:47.938ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2042d0bb-f968-44a8-a1df-483336b0146b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73392025-04-30T06:35:47.938ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
73402025-04-30T06:35:47.941ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54990b94-fb86-4e20-959d-bbef6276a88c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73412025-04-30T06:35:47.941ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
73422025-04-30T06:35:47.943ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2461603-05cb-4325-8506-2ce4eaca0f40", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73432025-04-30T06:35:47.943ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
73442025-04-30T06:35:47.944ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a704cc6-cc56-4f6d-aa88-475a79b4025c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73452025-04-30T06:35:47.945ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
73462025-04-30T06:35:47.946ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "290cc69e-970f-42ab-9d78-0a9c136100a7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73472025-04-30T06:35:47.946ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
73482025-04-30T06:35:47.947ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cf01f34-ee8b-4d87-8a87-fd1de3a7b2a0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73492025-04-30T06:35:47.950ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
73502025-04-30T06:35:47.950ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c32bb99-03ed-4bbc-b4e3-e1ba043a52a6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73512025-04-30T06:35:47.950ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
73522025-04-30T06:35:47.950ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "483447d2-f693-442a-9215-a579732aa45a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73532025-04-30T06:35:47.950ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
73542025-04-30T06:35:47.951ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "87a7c476-bb90-4c54-97ae-0b910b1dbf3c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73552025-04-30T06:35:47.952ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
73562025-04-30T06:35:47.954ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b07e0a22-fcf1-465c-9c2b-07af2dee5b0b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73572025-04-30T06:35:47.954ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
73582025-04-30T06:35:47.956ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2494f18-aa4b-405e-9a12-fca02da218be", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73592025-04-30T06:35:47.956ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
73602025-04-30T06:35:47.960ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee281f04-a140-4efb-b239-efc95c92d63c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73612025-04-30T06:35:47.960ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
73622025-04-30T06:35:47.962ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d12e18a-69f4-4df4-89b7-bd8f72a3e42b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73632025-04-30T06:35:47.963ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
73642025-04-30T06:35:47.965ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6f52dbf-c929-4b7a-ad33-bbcac203d05b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73652025-04-30T06:35:47.965ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
73662025-04-30T06:35:47.966ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fc5fcdf-9fc4-43b6-aae8-8ee93086095a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73672025-04-30T06:35:47.966ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
73682025-04-30T06:35:47.967ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5b08f70-b8e3-4761-81e5-a6411f17f2a3", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73692025-04-30T06:35:47.967ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
73702025-04-30T06:35:47.968ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05954941-c5a4-443b-b9b8-522123813e51", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73712025-04-30T06:35:47.968ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
73722025-04-30T06:35:47.971ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7c4162b-c355-4a57-9b61-576bfbbfa4d9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73732025-04-30T06:35:47.971ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
73742025-04-30T06:35:47.971ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ac9673-09ea-4261-aaf6-f8ffce48805e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73752025-04-30T06:35:47.971ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
73762025-04-30T06:35:47.972ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eae67809-58b3-459c-b2ec-b451e8e4d5b5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73772025-04-30T06:35:47.972ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
73782025-04-30T06:35:47.975ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb80ae51-65bd-4ef3-9367-31c821610111", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73792025-04-30T06:35:47.975ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
73802025-04-30T06:35:47.977ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db758596-93be-40c3-8cc0-82a1fbb46aed", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73812025-04-30T06:35:47.981ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
73822025-04-30T06:35:47.981ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
73832025-04-30T06:35:47.981ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
73842025-04-30T06:35:47.981ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73852025-04-30T06:35:47.981ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73862025-04-30T06:35:47.986ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73872025-04-30T06:35:47.986ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9239a16b-4d07-4ec8-86d1-4a2ca1a2318e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73882025-04-30T06:35:47.986ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
73892025-04-30T06:35:47.986ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "54d10196-588e-4e97-a423-7403515dec21", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
73902025-04-30T06:35:47.986ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
73912025-04-30T06:35:47.987ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
73922025-04-30T06:35:48.004ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
73932025-04-30T06:35:48.005ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73942025-04-30T06:35:48.005ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73952025-04-30T06:35:48.007ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73962025-04-30T06:35:48.009ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
73972025-04-30T06:35:48.009ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
73982025-04-30T06:35:48.011ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
73992025-04-30T06:35:48.013ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2d8a6da-b106-4d30-b8c1-efdae108c223", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74002025-04-30T06:35:48.013ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
74012025-04-30T06:35:48.015ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fabe22fa-c625-4616-90b8-4b5c7ae2f611", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74022025-04-30T06:35:48.015ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
74032025-04-30T06:35:48.017ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a30c540a-33c4-4709-b760-16ea7832bf9f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74042025-04-30T06:35:48.017ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
74052025-04-30T06:35:48.018ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d66746d8-6731-41a9-a95d-9f3d29b0c73a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74062025-04-30T06:35:48.018ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
74072025-04-30T06:35:48.020ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "039e1f26-d426-4e58-8175-f99ccca903fe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74082025-04-30T06:35:48.020ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74092025-04-30T06:35:48.020ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74102025-04-30T06:35:48.020ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74112025-04-30T06:35:48.020ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74122025-04-30T06:35:48.021ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74132025-04-30T06:35:48.023ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74142025-04-30T06:35:48.029ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74152025-04-30T06:35:48.029ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
74162025-04-30T06:35:48.029ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74172025-04-30T06:35:48.029ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74182025-04-30T06:35:48.029ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74192025-04-30T06:35:48.030ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74202025-04-30T06:35:48.033ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74212025-04-30T06:35:48.038ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
74222025-04-30T06:35:48.038ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74232025-04-30T06:35:48.038ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74242025-04-30T06:35:48.038ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74252025-04-30T06:35:48.102ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
74262025-04-30T06:35:48.102ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
74272025-04-30T06:35:48.105ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:54309 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
74282025-04-30T06:35:48.106ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:54309 req_id = 2a5adee7-0af5-4c68-bb37-ff1dfbd18d74 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74292025-04-30T06:35:48.107ZINFOtest_metrics (dropshot): request completed latency_us = 975 local_addr = [::1]:56237 method = GET remote_addr = [::1]:54309 req_id = 2a5adee7-0af5-4c68-bb37-ff1dfbd18d74 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
74302025-04-30T06:35:48.108ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a5adee7-0af5-4c68-bb37-ff1dfbd18d74", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74312025-04-30T06:35:48.112ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74322025-04-30T06:35:48.113ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
74332025-04-30T06:35:48.113ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
74342025-04-30T06:35:48.117ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74352025-04-30T06:35:48.117ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74362025-04-30T06:35:48.117ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74372025-04-30T06:35:48.117ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74382025-04-30T06:35:48.119ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74392025-04-30T06:35:48.125ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74402025-04-30T06:35:48.133ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74412025-04-30T06:35:48.134ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74422025-04-30T06:35:48.134ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74432025-04-30T06:35:48.136ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74442025-04-30T06:35:48.137ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
74452025-04-30T06:35:48.176ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
74462025-04-30T06:35:48.176ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
74472025-04-30T06:35:48.177ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 390.609033ms iteration = 22 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
74482025-04-30T06:35:48.177ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 23 reason = Signaled
74492025-04-30T06:35:48.177ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
74502025-04-30T06:35:48.177ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74512025-04-30T06:35:48.177ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74522025-04-30T06:35:48.177ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74532025-04-30T06:35:48.178ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74542025-04-30T06:35:48.181ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74552025-04-30T06:35:48.188ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
74562025-04-30T06:35:48.188ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
74572025-04-30T06:35:48.188ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
74582025-04-30T06:35:48.189ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
74592025-04-30T06:35:48.190ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
74602025-04-30T06:35:48.190ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
74612025-04-30T06:35:48.190ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74622025-04-30T06:35:48.190ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
74632025-04-30T06:35:48.192ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74642025-04-30T06:35:48.194ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
74652025-04-30T06:35:48.201ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
74662025-04-30T06:35:48.202ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74672025-04-30T06:35:48.202ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
74682025-04-30T06:35:48.202ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74692025-04-30T06:35:48.203ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
74702025-04-30T06:35:48.203ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
74712025-04-30T06:35:48.204ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 24 50 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 $P..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:50208 req_id = cf52c7d6-628c-42d1-99fd-444dd1b3683a
74722025-04-30T06:35:48.204ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9296,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50208 req_id = cf52c7d6-628c-42d1-99fd-444dd1b3683a
74732025-04-30T06:35:48.204ZDEBGtest_metrics (store): query key key = _mgs._tcp
74742025-04-30T06:35:48.204ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74752025-04-30T06:35:48.205ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50208 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = cf52c7d6-628c-42d1-99fd-444dd1b3683a
74762025-04-30T06:35:48.205ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
74772025-04-30T06:35:48.205ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
74782025-04-30T06:35:48.206ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
74792025-04-30T06:35:48.206ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3813, tv_nsec: 5405981 } })
74802025-04-30T06:35:48.206ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
74812025-04-30T06:35:48.207ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 43 6b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Ck..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65295 req_id = 07d9c3ae-0ab8-47bf-a0e8-125e68562e60
74822025-04-30T06:35:48.207ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17259,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65295 req_id = 07d9c3ae-0ab8-47bf-a0e8-125e68562e60
74832025-04-30T06:35:48.207ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
74842025-04-30T06:35:48.207ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65295 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 07d9c3ae-0ab8-47bf-a0e8-125e68562e60
74852025-04-30T06:35:48.271ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
74862025-04-30T06:35:48.272ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 56 0f 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c V............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60204 req_id = 8885dc71-3a7c-49df-a379-398ea0dd5211
74872025-04-30T06:35:48.272ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22031,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60204 req_id = 8885dc71-3a7c-49df-a379-398ea0dd5211
74882025-04-30T06:35:48.272ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
74892025-04-30T06:35:48.272ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22031, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60204 req_id = 8885dc71-3a7c-49df-a379-398ea0dd5211
74902025-04-30T06:35:48.272ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
74912025-04-30T06:35:48.272ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
74922025-04-30T06:35:48.272ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
74932025-04-30T06:35:48.275ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb6bf89a-35e4-4181-bf50-b4a28fe65843", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74942025-04-30T06:35:48.276ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
74952025-04-30T06:35:48.278ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc90dac5-5fcd-4f5b-8f53-d6f6cdf48eb2", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74962025-04-30T06:35:48.278ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
74972025-04-30T06:35:48.282ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "75d23361-cc24-4fff-8fb9-6a36badb08d1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
74982025-04-30T06:35:48.282ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
74992025-04-30T06:35:48.289ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51d8e5bb-a9ed-4ccf-9032-e7153fe1c745", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75002025-04-30T06:35:48.289ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
75012025-04-30T06:35:48.293ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7643054f-8dbc-45e9-92d6-978622da395b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75022025-04-30T06:35:48.294ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
75032025-04-30T06:35:48.313ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "01087b17-0666-4ce3-bd71-67afdf5a21df", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75042025-04-30T06:35:48.314ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
75052025-04-30T06:35:48.314ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
75062025-04-30T06:35:48.318ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "639bbc0e-c6cb-4fce-994a-076da157e752", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75072025-04-30T06:35:48.318ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
75082025-04-30T06:35:48.320ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "962c2da8-ea3c-4289-9b50-c640f01a15bc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75092025-04-30T06:35:48.320ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
75102025-04-30T06:35:48.321ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed7db71b-22ec-4cb3-98c4-935671baadb7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75112025-04-30T06:35:48.321ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
75122025-04-30T06:35:48.322ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26517694-e59e-4742-9715-7a6d5423b3e5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75132025-04-30T06:35:48.322ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
75142025-04-30T06:35:48.336ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b85c1a7-f904-44ee-b2b5-2ad2b4c7780c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75152025-04-30T06:35:48.337ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
75162025-04-30T06:35:48.349ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "542c8e86-cadf-482b-988a-0f24113337f7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75172025-04-30T06:35:48.349ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
75182025-04-30T06:35:48.367ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "03d870db-7ac4-4665-acec-27829ef79ce0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75192025-04-30T06:35:48.368ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
75202025-04-30T06:35:48.368ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75212025-04-30T06:35:48.368ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75222025-04-30T06:35:48.368ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75232025-04-30T06:35:48.368ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75242025-04-30T06:35:48.370ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75252025-04-30T06:35:48.381ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
75262025-04-30T06:35:48.387ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
75272025-04-30T06:35:48.387ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75282025-04-30T06:35:48.388ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75292025-04-30T06:35:48.388ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75302025-04-30T06:35:48.396ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75312025-04-30T06:35:48.396ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75322025-04-30T06:35:48.397ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75332025-04-30T06:35:48.398ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ec5bd0b-f5c8-4554-8307-0df37809c0c1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75342025-04-30T06:35:48.398ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
75352025-04-30T06:35:48.404ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "26b86e51-d1c7-4913-a084-b93bbfcda3a3", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75362025-04-30T06:35:48.404ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
75372025-04-30T06:35:48.407ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "90234bf5-c4f7-42fa-8c4b-c32c32c7d3b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75382025-04-30T06:35:48.407ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
75392025-04-30T06:35:48.409ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c35bacba-9404-4b28-a936-0cdbe0c7fd14", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75402025-04-30T06:35:48.409ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
75412025-04-30T06:35:48.413ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21789f00-f04e-4f86-96d0-b1ab482ec9c6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75422025-04-30T06:35:48.413ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
75432025-04-30T06:35:48.416ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "34545379-2164-4f86-b9d8-90d61f9a7fc0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75442025-04-30T06:35:48.416ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
75452025-04-30T06:35:48.417ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d79a6fb-944b-46bf-9d29-6c7dd9ba06d4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75462025-04-30T06:35:48.417ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
75472025-04-30T06:35:48.419ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b36caa87-e7a2-4705-b71b-9c8e64a2ff89", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75482025-04-30T06:35:48.419ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
75492025-04-30T06:35:48.423ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d941b9f-0d77-4933-9b72-169c2fb498c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75502025-04-30T06:35:48.424ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
75512025-04-30T06:35:48.427ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac167fc6-dd87-4b96-acc1-0fb3aa331cca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75522025-04-30T06:35:48.428ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
75532025-04-30T06:35:48.428ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
75542025-04-30T06:35:48.430ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "29e26ebf-97a3-4f08-86f6-9949f3447616", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75552025-04-30T06:35:48.430ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
75562025-04-30T06:35:48.434ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba409dca-5aff-4bc2-a4cb-853d7884dedc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75572025-04-30T06:35:48.434ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
75582025-04-30T06:35:48.437ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b1d8027-52d6-4835-bb0d-657834bf65e7", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75592025-04-30T06:35:48.437ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
75602025-04-30T06:35:48.439ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "16aafc76-630d-47d1-8318-d783385cb583", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75612025-04-30T06:35:48.439ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
75622025-04-30T06:35:48.441ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e116bc9-d528-43a7-9f3e-4fdfc97d3f0d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75632025-04-30T06:35:48.441ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
75642025-04-30T06:35:48.444ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20ec9c9e-aa67-4541-95fa-d1bc87c0838a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75652025-04-30T06:35:48.444ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
75662025-04-30T06:35:48.446ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f7b698e-3113-4bab-b58c-a9cca8ec655e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75672025-04-30T06:35:48.446ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
75682025-04-30T06:35:48.448ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c34daea9-821c-4c88-b53a-1f43fbb47206", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75692025-04-30T06:35:48.448ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
75702025-04-30T06:35:48.450ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3ee33ef-daff-40c8-b0a3-b539d98d1d1f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75712025-04-30T06:35:48.450ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
75722025-04-30T06:35:48.451ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d68449c0-4635-437f-88c0-a779f4c4017b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75732025-04-30T06:35:48.451ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
75742025-04-30T06:35:48.453ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f656464-939c-4aca-bc70-3ea1ee58f72a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75752025-04-30T06:35:48.453ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
75762025-04-30T06:35:48.455ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9af06b6a-3027-4d87-a655-f6746c87ad9b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75772025-04-30T06:35:48.455ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
75782025-04-30T06:35:48.457ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0a6d4f6-d336-45d3-a511-56de42c1a9aa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75792025-04-30T06:35:48.457ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
75802025-04-30T06:35:48.459ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63cec95c-8dd8-4fb3-bb6f-b62363f8fd6b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75812025-04-30T06:35:48.459ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
75822025-04-30T06:35:48.461ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8680d46f-e6f6-431e-b0b8-bd1ce05a45f2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75832025-04-30T06:35:48.461ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
75842025-04-30T06:35:48.463ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cb1e24b-f42c-4dc7-969e-c7cf3f61cd5d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75852025-04-30T06:35:48.463ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
75862025-04-30T06:35:48.465ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e0d13d35-5214-4579-a44a-a020533eb91d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75872025-04-30T06:35:48.465ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
75882025-04-30T06:35:48.476ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78c21f24-5ceb-4b9d-bde1-514dab15df24", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75892025-04-30T06:35:48.477ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
75902025-04-30T06:35:48.478ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c82d97a1-9fd7-42ef-b660-98f6d02bfdc4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75912025-04-30T06:35:48.478ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
75922025-04-30T06:35:48.479ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
75932025-04-30T06:35:48.479ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
75942025-04-30T06:35:48.479ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
75952025-04-30T06:35:48.479ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
75962025-04-30T06:35:48.480ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
75972025-04-30T06:35:48.481ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7a7f5de-4b38-4bf5-b33b-0f4756686ed6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
75982025-04-30T06:35:48.481ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
75992025-04-30T06:35:48.482ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7fb7040-bf28-47d8-955e-f01844fc4929", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
76002025-04-30T06:35:48.483ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
76012025-04-30T06:35:48.483ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76022025-04-30T06:35:48.490ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
76032025-04-30T06:35:48.490ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76042025-04-30T06:35:48.490ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76052025-04-30T06:35:48.490ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76062025-04-30T06:35:48.493ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61629ae6-1aea-4521-9d81-c72d7f1793e1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:47 GMT"} })
76072025-04-30T06:35:48.493ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76082025-04-30T06:35:48.493ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76092025-04-30T06:35:48.493ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76102025-04-30T06:35:48.493ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76112025-04-30T06:35:48.493ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76122025-04-30T06:35:48.494ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76132025-04-30T06:35:48.494ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76142025-04-30T06:35:48.495ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76152025-04-30T06:35:48.496ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76162025-04-30T06:35:48.500ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76172025-04-30T06:35:48.500ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
76182025-04-30T06:35:48.500ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76192025-04-30T06:35:48.500ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76202025-04-30T06:35:48.500ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76212025-04-30T06:35:48.500ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76222025-04-30T06:35:48.503ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76232025-04-30T06:35:48.507ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
76242025-04-30T06:35:48.507ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76252025-04-30T06:35:48.507ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76262025-04-30T06:35:48.508ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76272025-04-30T06:35:48.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76282025-04-30T06:35:48.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76292025-04-30T06:35:48.556ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
76302025-04-30T06:35:48.556ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
76312025-04-30T06:35:48.556ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:44759 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
76322025-04-30T06:35:48.558ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:44759 req_id = 156f90fa-d5e5-4d80-b7bc-7ed7f0ff3212 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76332025-04-30T06:35:48.558ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76342025-04-30T06:35:48.559ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76352025-04-30T06:35:48.559ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76362025-04-30T06:35:48.559ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
76372025-04-30T06:35:48.559ZINFOtest_metrics (dropshot): request completed latency_us = 1197 local_addr = [::1]:56237 method = GET remote_addr = [::1]:44759 req_id = 156f90fa-d5e5-4d80-b7bc-7ed7f0ff3212 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
76382025-04-30T06:35:48.560ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "156f90fa-d5e5-4d80-b7bc-7ed7f0ff3212", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
76392025-04-30T06:35:48.563ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76402025-04-30T06:35:48.563ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
76412025-04-30T06:35:48.563ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
76422025-04-30T06:35:48.566ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76432025-04-30T06:35:48.566ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76442025-04-30T06:35:48.566ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76452025-04-30T06:35:48.566ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76462025-04-30T06:35:48.568ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76472025-04-30T06:35:48.571ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76482025-04-30T06:35:48.571ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76492025-04-30T06:35:48.571ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
76502025-04-30T06:35:48.572ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
76512025-04-30T06:35:48.578ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
76522025-04-30T06:35:48.578ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
76532025-04-30T06:35:48.578ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76542025-04-30T06:35:48.583ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76552025-04-30T06:35:48.583ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76562025-04-30T06:35:48.583ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76572025-04-30T06:35:48.584ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76582025-04-30T06:35:48.588ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
76592025-04-30T06:35:48.588ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
76602025-04-30T06:35:48.596ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
76612025-04-30T06:35:48.613ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
76622025-04-30T06:35:48.614ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
76632025-04-30T06:35:48.614ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 437.082328ms iteration = 23 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
76642025-04-30T06:35:48.614ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 24 reason = Signaled
76652025-04-30T06:35:48.614ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
76662025-04-30T06:35:48.614ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76672025-04-30T06:35:48.614ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76682025-04-30T06:35:48.614ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76692025-04-30T06:35:48.615ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76702025-04-30T06:35:48.619ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
76712025-04-30T06:35:48.623ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
76722025-04-30T06:35:48.623ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
76732025-04-30T06:35:48.623ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
76742025-04-30T06:35:48.623ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
76752025-04-30T06:35:48.626ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
76762025-04-30T06:35:48.626ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
76772025-04-30T06:35:48.627ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: c7 ff 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57848 req_id = db064b57-a86d-40a0-9ecd-ab887f4736f2
76782025-04-30T06:35:48.627ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51199,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57848 req_id = db064b57-a86d-40a0-9ecd-ab887f4736f2
76792025-04-30T06:35:48.628ZDEBGtest_metrics (store): query key key = _mgs._tcp
76802025-04-30T06:35:48.628ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76812025-04-30T06:35:48.628ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57848 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = db064b57-a86d-40a0-9ecd-ab887f4736f2
76822025-04-30T06:35:48.628ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3813, tv_nsec: 427743611 } })
76832025-04-30T06:35:48.628ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
76842025-04-30T06:35:48.629ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 85 8f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60963 req_id = 68bd241e-d3e9-4d22-a0bf-4836ae6e65a2
76852025-04-30T06:35:48.629ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34191,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60963 req_id = 68bd241e-d3e9-4d22-a0bf-4836ae6e65a2
76862025-04-30T06:35:48.629ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
76872025-04-30T06:35:48.629ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60963 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 68bd241e-d3e9-4d22-a0bf-4836ae6e65a2
76882025-04-30T06:35:48.681ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
76892025-04-30T06:35:48.681ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
76902025-04-30T06:35:48.681ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
76912025-04-30T06:35:48.681ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
76922025-04-30T06:35:48.681ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
76932025-04-30T06:35:48.682ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
76942025-04-30T06:35:48.682ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
76952025-04-30T06:35:48.683ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: fd fd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:52541 req_id = 35b5e596-df4f-414b-babf-04e1535c184c
76962025-04-30T06:35:48.683ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 65021,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52541 req_id = 35b5e596-df4f-414b-babf-04e1535c184c
76972025-04-30T06:35:48.683ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
76982025-04-30T06:35:48.683ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 65021, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:52541 req_id = 35b5e596-df4f-414b-babf-04e1535c184c
76992025-04-30T06:35:48.683ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
77002025-04-30T06:35:48.683ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
77012025-04-30T06:35:48.683ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
77022025-04-30T06:35:48.685ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
77032025-04-30T06:35:48.693ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
77042025-04-30T06:35:48.693ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77052025-04-30T06:35:48.693ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
77062025-04-30T06:35:48.694ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77072025-04-30T06:35:48.695ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "80e6c5d7-0f56-458a-a6ad-ce22e769e2f1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77082025-04-30T06:35:48.696ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
77092025-04-30T06:35:48.697ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77102025-04-30T06:35:48.697ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
77112025-04-30T06:35:48.698ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77122025-04-30T06:35:48.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "230d66aa-b632-4325-b513-9c27c8cd7c1c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77132025-04-30T06:35:48.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
77142025-04-30T06:35:48.703ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a12e89fe-a38c-4707-959c-0607fe3d9b75", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77152025-04-30T06:35:48.703ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
77162025-04-30T06:35:48.706ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3dc0c94-4726-4036-80b9-0d8fd3b4b51b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77172025-04-30T06:35:48.706ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
77182025-04-30T06:35:48.710ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "88d4c30d-b568-4714-9a29-5dfb692ac9a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77192025-04-30T06:35:48.710ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
77202025-04-30T06:35:48.713ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17d9d0a4-1ee2-4e58-bd6b-088287f8888a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77212025-04-30T06:35:48.713ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
77222025-04-30T06:35:48.717ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d3dace5-3fe0-4bbf-9dcc-c9e092d0e70e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77232025-04-30T06:35:48.717ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
77242025-04-30T06:35:48.721ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a45d15a0-9359-433c-aa92-ffe115a5ea75", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77252025-04-30T06:35:48.721ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
77262025-04-30T06:35:48.724ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fd4f79f-678d-4281-9e8b-bb81e73f5dab", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77272025-04-30T06:35:48.724ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
77282025-04-30T06:35:48.727ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a6e7762-fd43-420a-b181-874467bf552a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77292025-04-30T06:35:48.727ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
77302025-04-30T06:35:48.727ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
77312025-04-30T06:35:48.729ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9eba9c43-6410-407d-bddd-336bc3118131", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77322025-04-30T06:35:48.729ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
77332025-04-30T06:35:48.731ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c7b165e9-30d7-4488-8493-c368db31bc7b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77342025-04-30T06:35:48.731ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
77352025-04-30T06:35:48.733ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0706dcc-baf0-4f3a-9b98-4895a22f4549", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77362025-04-30T06:35:48.733ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
77372025-04-30T06:35:48.735ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f63f3d9-bd29-4bd3-9e34-29e5d69e831e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77382025-04-30T06:35:48.735ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
77392025-04-30T06:35:48.737ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccdf2656-6831-4285-ad42-67b11a835aa2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77402025-04-30T06:35:48.737ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
77412025-04-30T06:35:48.739ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c16e39da-46f6-4cd0-9135-71d7d7f0a8a0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77422025-04-30T06:35:48.739ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
77432025-04-30T06:35:48.741ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "516ca7a5-6668-449d-8ed8-c276edf7e24f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77442025-04-30T06:35:48.742ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
77452025-04-30T06:35:48.744ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdfc3226-f710-407a-b736-2b5a684c9528", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77462025-04-30T06:35:48.744ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
77472025-04-30T06:35:48.747ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2739513c-2a0b-4ba2-98dd-21600dc536eb", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77482025-04-30T06:35:48.747ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
77492025-04-30T06:35:48.748ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "085594e2-bba0-4ab8-b760-97380c1796d9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77502025-04-30T06:35:48.748ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
77512025-04-30T06:35:48.749ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bea42d1-a6dd-4397-9b91-9dd059f3422b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77522025-04-30T06:35:48.749ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
77532025-04-30T06:35:48.750ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d90e57ff-8077-4b20-ad60-211a927670f0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77542025-04-30T06:35:48.750ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
77552025-04-30T06:35:48.751ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c112e65-7234-454b-81b3-40dae8ba9765", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77562025-04-30T06:35:48.751ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
77572025-04-30T06:35:48.752ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "555b0805-041f-4bd9-bb2a-5a52e293237e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77582025-04-30T06:35:48.752ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
77592025-04-30T06:35:48.754ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4ee280c-304b-4379-9c7c-0299d028690c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77602025-04-30T06:35:48.754ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
77612025-04-30T06:35:48.756ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be8260c7-8bac-449f-b961-be98ec3424d9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77622025-04-30T06:35:48.756ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
77632025-04-30T06:35:48.757ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a72abde-87ae-4eed-aff0-291ed8fa9f91", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77642025-04-30T06:35:48.758ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
77652025-04-30T06:35:48.759ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ca28749-8297-476f-9b86-59fc072895c9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77662025-04-30T06:35:48.759ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
77672025-04-30T06:35:48.761ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "954989d6-e0ab-4bc3-af6e-bdf155b82daf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77682025-04-30T06:35:48.761ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
77692025-04-30T06:35:48.763ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f71de6a-c8d4-4202-96a5-f74016cbc847", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77702025-04-30T06:35:48.763ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
77712025-04-30T06:35:48.763ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "667e5227-ca88-4502-afd6-4af95f665228", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77722025-04-30T06:35:48.764ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
77732025-04-30T06:35:48.765ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c667880-aed5-4eeb-9ce9-603c133bca5e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77742025-04-30T06:35:48.765ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
77752025-04-30T06:35:48.766ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd4a175d-5ebd-43c0-b60d-560c01d24e8d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77762025-04-30T06:35:48.766ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
77772025-04-30T06:35:48.767ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b36d45d-e508-424d-a4c3-db81f778f29b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77782025-04-30T06:35:48.767ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
77792025-04-30T06:35:48.767ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c83aad4-0e4d-4b9e-9bff-545912c9d099", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77802025-04-30T06:35:48.768ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
77812025-04-30T06:35:48.769ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "911eeb48-12af-45c2-8162-0547d9aae235", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77822025-04-30T06:35:48.769ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
77832025-04-30T06:35:48.771ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d6dde0d-c02f-4ead-b7f6-31d4cba8fecb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77842025-04-30T06:35:48.771ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
77852025-04-30T06:35:48.772ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74ed554e-7f0f-4132-8a86-ed6e147185de", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77862025-04-30T06:35:48.772ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
77872025-04-30T06:35:48.774ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48f2ec5a-9aba-4255-9775-257c35afdf04", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77882025-04-30T06:35:48.774ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
77892025-04-30T06:35:48.776ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe1a87e-79ba-4232-b291-03ecbd6e7249", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77902025-04-30T06:35:48.776ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
77912025-04-30T06:35:48.778ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76052450-49e5-4a62-9d8c-d190a77ab44d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77922025-04-30T06:35:48.778ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
77932025-04-30T06:35:48.779ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
77942025-04-30T06:35:48.779ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
77952025-04-30T06:35:48.779ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
77962025-04-30T06:35:48.779ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
77972025-04-30T06:35:48.780ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
77982025-04-30T06:35:48.781ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "68e9dfc0-1539-4e81-8fe8-9db4c5cacb9c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
77992025-04-30T06:35:48.781ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
78002025-04-30T06:35:48.783ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78012025-04-30T06:35:48.791ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78022025-04-30T06:35:48.791ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78032025-04-30T06:35:48.791ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78042025-04-30T06:35:48.791ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78052025-04-30T06:35:48.791ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "728c77b5-da3e-44c6-8a1d-37e111945fcc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
78062025-04-30T06:35:48.791ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
78072025-04-30T06:35:48.792ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "132bbc1b-b7f1-4f34-a870-80554d01c55f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
78082025-04-30T06:35:48.792ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
78092025-04-30T06:35:48.793ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78102025-04-30T06:35:48.793ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78112025-04-30T06:35:48.793ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78122025-04-30T06:35:48.796ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b365201f-98aa-404b-b1eb-e2ff0dca3774", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
78132025-04-30T06:35:48.796ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78142025-04-30T06:35:48.796ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78152025-04-30T06:35:48.796ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78162025-04-30T06:35:48.796ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78172025-04-30T06:35:48.796ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78182025-04-30T06:35:48.799ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78192025-04-30T06:35:48.803ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78202025-04-30T06:35:48.803ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
78212025-04-30T06:35:48.803ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78222025-04-30T06:35:48.803ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78232025-04-30T06:35:48.803ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78242025-04-30T06:35:48.804ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78252025-04-30T06:35:48.807ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78262025-04-30T06:35:48.812ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
78272025-04-30T06:35:48.812ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78282025-04-30T06:35:48.812ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78292025-04-30T06:35:48.813ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78302025-04-30T06:35:48.860ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
78312025-04-30T06:35:48.861ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
78322025-04-30T06:35:48.861ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:44839 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
78332025-04-30T06:35:48.862ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:44839 req_id = 8c688f62-d1b1-4fa9-9c9f-f2b63fe58f7d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
78342025-04-30T06:35:48.863ZINFOtest_metrics (dropshot): request completed latency_us = 533 local_addr = [::1]:56237 method = GET remote_addr = [::1]:44839 req_id = 8c688f62-d1b1-4fa9-9c9f-f2b63fe58f7d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
78352025-04-30T06:35:48.863ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c688f62-d1b1-4fa9-9c9f-f2b63fe58f7d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
78362025-04-30T06:35:48.866ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
78372025-04-30T06:35:48.866ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
78382025-04-30T06:35:48.866ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
78392025-04-30T06:35:48.869ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
78402025-04-30T06:35:48.869ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78412025-04-30T06:35:48.869ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78422025-04-30T06:35:48.869ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78432025-04-30T06:35:48.870ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78442025-04-30T06:35:48.875ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78452025-04-30T06:35:48.878ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78462025-04-30T06:35:48.878ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78472025-04-30T06:35:48.878ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78482025-04-30T06:35:48.879ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78492025-04-30T06:35:48.895ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
78502025-04-30T06:35:48.909ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
78512025-04-30T06:35:48.909ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
78522025-04-30T06:35:48.909ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 295.701565ms iteration = 24 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
78532025-04-30T06:35:48.909ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 25 reason = Signaled
78542025-04-30T06:35:48.910ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
78552025-04-30T06:35:48.910ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78562025-04-30T06:35:48.910ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78572025-04-30T06:35:48.910ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78582025-04-30T06:35:48.910ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78592025-04-30T06:35:48.912ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78602025-04-30T06:35:48.916ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
78612025-04-30T06:35:48.916ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
78622025-04-30T06:35:48.916ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
78632025-04-30T06:35:48.916ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
78642025-04-30T06:35:48.919ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
78652025-04-30T06:35:48.919ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
78662025-04-30T06:35:48.920ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: f3 11 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62491 req_id = 0d2c7ac5-2a1a-4574-8ba2-a407834643e7
78672025-04-30T06:35:48.920ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62225,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62491 req_id = 0d2c7ac5-2a1a-4574-8ba2-a407834643e7
78682025-04-30T06:35:48.921ZDEBGtest_metrics (store): query key key = _mgs._tcp
78692025-04-30T06:35:48.921ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78702025-04-30T06:35:48.921ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62491 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 0d2c7ac5-2a1a-4574-8ba2-a407834643e7
78712025-04-30T06:35:48.921ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3813, tv_nsec: 720235442 } })
78722025-04-30T06:35:48.921ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
78732025-04-30T06:35:48.921ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 76 1f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e v...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57397 req_id = 13e49d8f-1817-4807-b950-4219dfe6b508
78742025-04-30T06:35:48.921ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30239,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57397 req_id = 13e49d8f-1817-4807-b950-4219dfe6b508
78752025-04-30T06:35:48.922ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
78762025-04-30T06:35:48.922ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57397 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 13e49d8f-1817-4807-b950-4219dfe6b508
78772025-04-30T06:35:48.973ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
78782025-04-30T06:35:48.973ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
78792025-04-30T06:35:48.973ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
78802025-04-30T06:35:48.973ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78812025-04-30T06:35:48.973ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78822025-04-30T06:35:48.973ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78832025-04-30T06:35:48.973ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: d9 40 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .@..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61285 req_id = 66185f20-37e0-4d95-98c2-9e94f38d4189
78842025-04-30T06:35:48.973ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 55616,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61285 req_id = 66185f20-37e0-4d95-98c2-9e94f38d4189
78852025-04-30T06:35:48.973ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
78862025-04-30T06:35:48.973ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 55616, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61285 req_id = 66185f20-37e0-4d95-98c2-9e94f38d4189
78872025-04-30T06:35:48.974ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
78882025-04-30T06:35:48.974ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
78892025-04-30T06:35:48.974ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
78902025-04-30T06:35:48.974ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
78912025-04-30T06:35:48.979ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
78922025-04-30T06:35:48.979ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78932025-04-30T06:35:48.979ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78942025-04-30T06:35:48.980ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
78952025-04-30T06:35:48.984ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3669c109-73c6-4e54-ac3e-ae8de2e9cbd5", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
78962025-04-30T06:35:48.984ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
78972025-04-30T06:35:48.984ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
78982025-04-30T06:35:48.985ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
78992025-04-30T06:35:48.985ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79002025-04-30T06:35:48.986ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "65ebd140-5e63-4c8c-ba1d-fd08caae30a6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79012025-04-30T06:35:48.986ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
79022025-04-30T06:35:48.988ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2253355e-bc73-4695-bdbf-9ce9c17700d0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79032025-04-30T06:35:48.989ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
79042025-04-30T06:35:49.002ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32ade4bb-3c74-43d3-abb1-4cb3c7f391b6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79052025-04-30T06:35:49.002ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
79062025-04-30T06:35:49.006ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82b94d63-7ec2-46e4-a90e-8ca606a07b0b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79072025-04-30T06:35:49.006ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
79082025-04-30T06:35:49.010ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e81739d7-be61-4f50-ac79-31d99d37ed25", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79092025-04-30T06:35:49.011ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
79102025-04-30T06:35:49.015ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d11a9277-a509-4227-b5e4-20293ba6c50c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79112025-04-30T06:35:49.015ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
79122025-04-30T06:35:49.019ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ea31985-ea8b-492f-b1e0-f54a2a060a22", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79132025-04-30T06:35:49.020ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
79142025-04-30T06:35:49.022ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cb67a1b-7e6e-4b38-ae5d-cae042cdd1d7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79152025-04-30T06:35:49.022ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
79162025-04-30T06:35:49.025ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e9f0029-6e85-4600-ae58-ae1cfa387ca1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79172025-04-30T06:35:49.025ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
79182025-04-30T06:35:49.025ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
79192025-04-30T06:35:49.029ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd939824-7a25-4caa-831b-dda33726e3e5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79202025-04-30T06:35:49.030ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
79212025-04-30T06:35:49.032ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0ae2455-a82b-4deb-89ad-898a301fdd4e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79222025-04-30T06:35:49.032ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
79232025-04-30T06:35:49.035ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d18a3f96-03a7-4482-a4e9-82cd14c6bc89", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79242025-04-30T06:35:49.035ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
79252025-04-30T06:35:49.038ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5f8484d-1e10-49b1-a3fa-575839de9a44", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79262025-04-30T06:35:49.038ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
79272025-04-30T06:35:49.041ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "986c41de-3e89-45a9-94b4-a1f3d6ce99e2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79282025-04-30T06:35:49.041ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
79292025-04-30T06:35:49.044ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9d10327-6398-4b93-a3d0-180d57ac9eeb", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79302025-04-30T06:35:49.044ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
79312025-04-30T06:35:49.047ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d34ab2b2-6905-4dc4-9247-907450d70ec4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79322025-04-30T06:35:49.047ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
79332025-04-30T06:35:49.049ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77a48251-c5a2-4ddb-bcec-55705963dffd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79342025-04-30T06:35:49.049ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
79352025-04-30T06:35:49.052ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb0c2af9-1ea3-4002-98b6-238c6fc99608", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79362025-04-30T06:35:49.052ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
79372025-04-30T06:35:49.053ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8388495e-d72e-4f1d-81e7-f0d5f5e76ecd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79382025-04-30T06:35:49.053ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
79392025-04-30T06:35:49.055ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ac6a80d-d7dc-42e3-bb6e-20c70224ac37", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79402025-04-30T06:35:49.056ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
79412025-04-30T06:35:49.058ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "662d4635-ffa9-4f7e-b020-fdf511cf1cf1", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79422025-04-30T06:35:49.058ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
79432025-04-30T06:35:49.059ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3bccbec-703f-4b19-84f1-5c23cf183db5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79442025-04-30T06:35:49.059ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
79452025-04-30T06:35:49.070ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d268c1a6-78f6-4cd6-935e-e93e8539134b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79462025-04-30T06:35:49.072ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
79472025-04-30T06:35:49.075ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48c5f107-1bc4-45fb-a042-b05b1033b4f3", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79482025-04-30T06:35:49.075ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
79492025-04-30T06:35:49.076ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
79502025-04-30T06:35:49.076ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
79512025-04-30T06:35:49.076ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79522025-04-30T06:35:49.076ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
79532025-04-30T06:35:49.077ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79542025-04-30T06:35:49.080ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5594a8e-c8d6-416e-8cbb-36011d3adc80", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79552025-04-30T06:35:49.080ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
79562025-04-30T06:35:49.081ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
79572025-04-30T06:35:49.089ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
79582025-04-30T06:35:49.090ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79592025-04-30T06:35:49.090ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
79602025-04-30T06:35:49.091ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79612025-04-30T06:35:49.094ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
79622025-04-30T06:35:49.094ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
79632025-04-30T06:35:49.095ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
79642025-04-30T06:35:49.101ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "60cf7cbd-3894-4eda-9060-02c51f4da49a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79652025-04-30T06:35:49.101ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
79662025-04-30T06:35:49.114ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dba9d2c1-9130-4541-aaa3-3bbef740e7b9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79672025-04-30T06:35:49.114ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
79682025-04-30T06:35:49.120ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "606b829c-c917-486e-b297-6da99fd25068", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79692025-04-30T06:35:49.120ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
79702025-04-30T06:35:49.126ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3b8e5190-7940-4fcc-be7a-a356ff6dc8f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79712025-04-30T06:35:49.127ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
79722025-04-30T06:35:49.129ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "df7a54f1-d8f5-4a2f-8519-a8a98da4358d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79732025-04-30T06:35:49.129ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
79742025-04-30T06:35:49.130ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9027fb4f-fdf8-4415-8746-6a00bf542a5e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79752025-04-30T06:35:49.131ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
79762025-04-30T06:35:49.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b27e8a99-844c-46da-b077-6a2eb0c292a9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79772025-04-30T06:35:49.133ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
79782025-04-30T06:35:49.137ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
79792025-04-30T06:35:49.138ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "01d867bb-c259-4cd3-b605-38cc4257b71b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79802025-04-30T06:35:49.138ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
79812025-04-30T06:35:49.142ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "944341f9-202a-4c50-a71e-46380beb9f20", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79822025-04-30T06:35:49.142ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
79832025-04-30T06:35:49.149ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "22e51427-5ee4-45fc-84d3-3a2eb33c3b19", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79842025-04-30T06:35:49.150ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
79852025-04-30T06:35:49.154ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ea37384-1fda-4589-8128-3aed83e991dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79862025-04-30T06:35:49.159ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
79872025-04-30T06:35:49.163ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7e6c510-b651-477e-bb0f-7c67ec2e088a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79882025-04-30T06:35:49.163ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
79892025-04-30T06:35:49.172ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b60f6657-98b6-4e92-af90-2c28ced80e5e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79902025-04-30T06:35:49.174ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
79912025-04-30T06:35:49.178ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1862e913-2e42-423d-aa66-3eb3de1b0ab9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79922025-04-30T06:35:49.179ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
79932025-04-30T06:35:49.183ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "055afa8c-6570-43a7-a83e-800121ae3984", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79942025-04-30T06:35:49.183ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
79952025-04-30T06:35:49.186ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2a03858-ee87-4904-873c-3a15e8f3fbcd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79962025-04-30T06:35:49.186ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
79972025-04-30T06:35:49.188ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "04401613-d415-4331-bd12-215c5d86a50b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
79982025-04-30T06:35:49.189ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
79992025-04-30T06:35:49.189ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80002025-04-30T06:35:49.189ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80012025-04-30T06:35:49.189ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80022025-04-30T06:35:49.189ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80032025-04-30T06:35:49.190ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80042025-04-30T06:35:49.193ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7768e82-8024-441d-886d-de1e3093e694", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
80052025-04-30T06:35:49.193ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
80062025-04-30T06:35:49.195ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80072025-04-30T06:35:49.202ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
80082025-04-30T06:35:49.202ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80092025-04-30T06:35:49.202ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80102025-04-30T06:35:49.203ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80112025-04-30T06:35:49.215ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27046cad-acf5-4fc0-ac93-6b5fa97ae20e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
80122025-04-30T06:35:49.215ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80132025-04-30T06:35:49.215ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80142025-04-30T06:35:49.216ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80152025-04-30T06:35:49.216ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80162025-04-30T06:35:49.217ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80172025-04-30T06:35:49.217ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
80182025-04-30T06:35:49.218ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
80192025-04-30T06:35:49.218ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
80202025-04-30T06:35:49.220ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80212025-04-30T06:35:49.225ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80222025-04-30T06:35:49.225ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
80232025-04-30T06:35:49.225ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80242025-04-30T06:35:49.225ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80252025-04-30T06:35:49.225ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80262025-04-30T06:35:49.226ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80272025-04-30T06:35:49.228ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80282025-04-30T06:35:49.232ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
80292025-04-30T06:35:49.232ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80302025-04-30T06:35:49.232ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80312025-04-30T06:35:49.233ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80322025-04-30T06:35:49.291ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
80332025-04-30T06:35:49.291ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
80342025-04-30T06:35:49.292ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:41657 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
80352025-04-30T06:35:49.293ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:41657 req_id = e2853274-2795-42e1-b219-0b21c69783a9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80362025-04-30T06:35:49.294ZINFOtest_metrics (dropshot): request completed latency_us = 830 local_addr = [::1]:56237 method = GET remote_addr = [::1]:41657 req_id = e2853274-2795-42e1-b219-0b21c69783a9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
80372025-04-30T06:35:49.294ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2853274-2795-42e1-b219-0b21c69783a9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
80382025-04-30T06:35:49.297ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80392025-04-30T06:35:49.297ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
80402025-04-30T06:35:49.297ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
80412025-04-30T06:35:49.299ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80422025-04-30T06:35:49.299ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80432025-04-30T06:35:49.299ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80442025-04-30T06:35:49.299ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80452025-04-30T06:35:49.300ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80462025-04-30T06:35:49.301ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
80472025-04-30T06:35:49.301ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
80482025-04-30T06:35:49.303ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5b1347c7-b790-4d69-922f-94f14343c2df uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
80492025-04-30T06:35:49.303ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5b1347c7-b790-4d69-922f-94f14343c2df resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
80502025-04-30T06:35:49.303ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5b1347c7-b790-4d69-922f-94f14343c2df roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
80512025-04-30T06:35:49.304ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5b1347c7-b790-4d69-922f-94f14343c2df resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
80522025-04-30T06:35:49.304ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80532025-04-30T06:35:49.309ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80542025-04-30T06:35:49.310ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80552025-04-30T06:35:49.310ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80562025-04-30T06:35:49.311ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80572025-04-30T06:35:49.314ZINFOtest_metrics (dropshot_internal): request completed latency_us = 11519 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 5b1347c7-b790-4d69-922f-94f14343c2df response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
80582025-04-30T06:35:49.314ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b1347c7-b790-4d69-922f-94f14343c2df", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
80592025-04-30T06:35:49.315ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
80602025-04-30T06:35:49.315ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = d41ebc89-e098-4139-88c5-c322112392fe uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
80612025-04-30T06:35:49.316ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = d41ebc89-e098-4139-88c5-c322112392fe resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
80622025-04-30T06:35:49.316ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = d41ebc89-e098-4139-88c5-c322112392fe roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
80632025-04-30T06:35:49.316ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = d41ebc89-e098-4139-88c5-c322112392fe resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
80642025-04-30T06:35:49.319ZINFOtest_metrics (dropshot_internal): request completed latency_us = 3959 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = d41ebc89-e098-4139-88c5-c322112392fe response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
80652025-04-30T06:35:49.320ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "d41ebc89-e098-4139-88c5-c322112392fe", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
80662025-04-30T06:35:49.320ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
80672025-04-30T06:35:49.320ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
80682025-04-30T06:35:49.320ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
80692025-04-30T06:35:49.328ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
80702025-04-30T06:35:49.350ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
80712025-04-30T06:35:49.350ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
80722025-04-30T06:35:49.351ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 441.048511ms iteration = 25 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
80732025-04-30T06:35:49.351ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 26 reason = Signaled
80742025-04-30T06:35:49.351ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
80752025-04-30T06:35:49.351ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
80762025-04-30T06:35:49.351ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80772025-04-30T06:35:49.351ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80782025-04-30T06:35:49.352ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80792025-04-30T06:35:49.358ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
80802025-04-30T06:35:49.363ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
80812025-04-30T06:35:49.363ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
80822025-04-30T06:35:49.363ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
80832025-04-30T06:35:49.364ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
80842025-04-30T06:35:49.373ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
80852025-04-30T06:35:49.373ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
80862025-04-30T06:35:49.375ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 7f e0 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:64231 req_id = 4cd28d1f-a65c-4ea8-8dd1-6a8820540dce
80872025-04-30T06:35:49.375ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32736,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64231 req_id = 4cd28d1f-a65c-4ea8-8dd1-6a8820540dce
80882025-04-30T06:35:49.376ZDEBGtest_metrics (store): query key key = _mgs._tcp
80892025-04-30T06:35:49.376ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80902025-04-30T06:35:49.376ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64231 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4cd28d1f-a65c-4ea8-8dd1-6a8820540dce
80912025-04-30T06:35:49.377ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3814, tv_nsec: 176312746 } })
80922025-04-30T06:35:49.377ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
80932025-04-30T06:35:49.377ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 70 74 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e pt..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54082 req_id = 0f4a1fba-8dab-466f-be3f-61795a8a3382
80942025-04-30T06:35:49.378ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28788,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54082 req_id = 0f4a1fba-8dab-466f-be3f-61795a8a3382
80952025-04-30T06:35:49.378ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
80962025-04-30T06:35:49.378ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54082 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 0f4a1fba-8dab-466f-be3f-61795a8a3382
80972025-04-30T06:35:49.442ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
80982025-04-30T06:35:49.442ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
80992025-04-30T06:35:49.443ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
81002025-04-30T06:35:49.443ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81012025-04-30T06:35:49.443ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81022025-04-30T06:35:49.445ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81032025-04-30T06:35:49.445ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
81042025-04-30T06:35:49.445ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
81052025-04-30T06:35:49.446ZINFOtest_metrics (dropshot): accepted connection file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:1025 local_addr = 127.0.0.1:65131 remote_addr = 127.0.0.1:48235
81062025-04-30T06:35:49.448ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2c da 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ,............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63916 req_id = e415f0dc-ef1d-4e0f-bc6f-5d7442c9cc8d
81072025-04-30T06:35:49.448ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11482,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63916 req_id = e415f0dc-ef1d-4e0f-bc6f-5d7442c9cc8d
81082025-04-30T06:35:49.449ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
81092025-04-30T06:35:49.449ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 11482, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63916 req_id = e415f0dc-ef1d-4e0f-bc6f-5d7442c9cc8d
81102025-04-30T06:35:49.450ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
81112025-04-30T06:35:49.450ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
81122025-04-30T06:35:49.450ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
81132025-04-30T06:35:49.450ZTRACtest_metrics (dropshot): incoming request local_addr = 127.0.0.1:65131 method = GET remote_addr = 127.0.0.1:48235 req_id = 99321261-9028-4688-aec7-0c35fb1be7ef uri = /588db90d-5640-4be3-9102-2ebf0af0c9a3
81142025-04-30T06:35:49.461ZINFOtest_metrics (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 13727 local_addr = 127.0.0.1:65131 method = GET remote_addr = 127.0.0.1:48235 req_id = 99321261-9028-4688-aec7-0c35fb1be7ef response_code = 200 uri = /588db90d-5640-4be3-9102-2ebf0af0c9a3
81152025-04-30T06:35:49.463ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
81162025-04-30T06:35:49.479ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
81172025-04-30T06:35:49.479ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
81182025-04-30T06:35:49.479ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
81192025-04-30T06:35:49.480ZDEBGtest_metrics (results-sink): inserting 20 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81202025-04-30T06:35:49.480ZDEBGtest_metrics (results-sink-cluster): inserting 20 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81212025-04-30T06:35:49.480ZDEBGtest_metrics (clickhouse-client): unrolling 20 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81222025-04-30T06:35:49.480ZDEBGtest_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81232025-04-30T06:35:49.480ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('integration_target:integration_metric') FORMAT Native;
81242025-04-30T06:35:49.481ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
81252025-04-30T06:35:49.493ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
81262025-04-30T06:35:49.493ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81272025-04-30T06:35:49.493ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81282025-04-30T06:35:49.498ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81292025-04-30T06:35:49.502ZTRACtest_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81302025-04-30T06:35:49.502ZDEBGtest_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e schema = TimeseriesSchema { timeseries_name: TimeseriesName("http_service:request_latency_histogram"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }, FieldSchema { name: "name", field_type: String, source: Target, description: "" }, FieldSchema { name: "operation_id", field_type: String, source: Metric, description: "" }, FieldSchema { name: "status_code", field_type: U16, source: Metric, description: "" }}, datum_type: HistogramU64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:35:49.480848328Z } timeseries_name = http_service:request_latency_histogram
81312025-04-30T06:35:49.510ZDEBGtest_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81322025-04-30T06:35:49.510ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
81332025-04-30T06:35:49.511ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
81342025-04-30T06:35:49.511ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
81352025-04-30T06:35:49.511ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
81362025-04-30T06:35:49.513ZTRACtest_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81372025-04-30T06:35:49.513ZDEBGtest_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:cpus_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:35:49.510699275Z } timeseries_name = collection_target:cpus_provisioned
81382025-04-30T06:35:49.514ZDEBGtest_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81392025-04-30T06:35:49.514ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
81402025-04-30T06:35:49.514ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81412025-04-30T06:35:49.514ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ed156bf-e796-4efc-98c6-531096090ad7", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81422025-04-30T06:35:49.515ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
81432025-04-30T06:35:49.515ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81442025-04-30T06:35:49.518ZTRACtest_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81452025-04-30T06:35:49.518ZDEBGtest_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:ram_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:35:49.514358756Z } timeseries_name = collection_target:ram_provisioned
81462025-04-30T06:35:49.519ZDEBGtest_metrics (clickhouse-client): retrieving timeseries schema from database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81472025-04-30T06:35:49.519ZTRACtest_metrics (clickhouse-client): executing SQL query collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e sql = SELECT * FROM oximeter.timeseries_schema WHERE timeseries_name NOT IN ('collection_target:cpus_provisioned', 'collection_target:ram_provisioned', 'http_service:request_latency_histogram', 'integration_target:integration_metric') FORMAT Native;
81482025-04-30T06:35:49.519ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81492025-04-30T06:35:49.520ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81502025-04-30T06:35:49.520ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81512025-04-30T06:35:49.520ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
81522025-04-30T06:35:49.521ZTRACtest_metrics (clickhouse-client): no new timeseries schema in database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81532025-04-30T06:35:49.522ZDEBGtest_metrics (clickhouse-client): new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e schema = TimeseriesSchema { timeseries_name: TimeseriesName("collection_target:virtual_disk_space_provisioned"), description: TimeseriesDescription { target: "", metric: "" }, field_schema: {FieldSchema { name: "id", field_type: Uuid, source: Target, description: "" }}, datum_type: I64, version: 1, authz_scope: Fleet, units: Count, created: 2025-04-30T06:35:49.519210332Z } timeseries_name = collection_target:virtual_disk_space_provisioned
81542025-04-30T06:35:49.528ZDEBGtest_metrics (clickhouse-client): inserting 4 new timeseries schema collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81552025-04-30T06:35:49.528ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 6 n_rows = 4 sql = INSERT INTO oximeter.timeseries_schema FORMAT Native
81562025-04-30T06:35:49.528ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdd4b610-6763-4208-a71c-27509b6fcd41", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81572025-04-30T06:35:49.529ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
81582025-04-30T06:35:49.533ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e9ae062-a337-4067-8b3f-e1bc0ba154ee", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81592025-04-30T06:35:49.533ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
81602025-04-30T06:35:49.534ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 16 sql = INSERT INTO oximeter.fields_string FORMAT Native
81612025-04-30T06:35:49.536ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "383d53fc-6037-4e46-93df-be6321e668c1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81622025-04-30T06:35:49.536ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
81632025-04-30T06:35:49.538ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 16 table_name = fields_string
81642025-04-30T06:35:49.538ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 8 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
81652025-04-30T06:35:49.539ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f55b362-674e-43bc-9e24-d925d3408020", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81662025-04-30T06:35:49.540ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
81672025-04-30T06:35:49.542ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 8 table_name = fields_u16
81682025-04-30T06:35:49.542ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 20 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
81692025-04-30T06:35:49.544ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94322217-4c97-4a26-ba6a-02d993741d21", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81702025-04-30T06:35:49.545ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
81712025-04-30T06:35:49.546ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 20 table_name = fields_uuid
81722025-04-30T06:35:49.546ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 19 n_rows = 8 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
81732025-04-30T06:35:49.549ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d251c4a-2664-4a46-af60-30586fd9d87c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81742025-04-30T06:35:49.549ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
81752025-04-30T06:35:49.551ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3f276bd-7e8c-4ad4-b8ba-80cd4c968c79", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81762025-04-30T06:35:49.552ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
81772025-04-30T06:35:49.553ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 8 table_name = measurements_histogramu64
81782025-04-30T06:35:49.553ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 12 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
81792025-04-30T06:35:49.555ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
81802025-04-30T06:35:49.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc413a7e-68eb-4547-b022-427ec325fdee", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:48 GMT"} })
81812025-04-30T06:35:49.556ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
81822025-04-30T06:35:49.557ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 12 table_name = measurements_i64
81832025-04-30T06:35:49.557ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81842025-04-30T06:35:49.559ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91cea95d-8240-4473-b9e1-79dee2bbe527", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
81852025-04-30T06:35:49.559ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
81862025-04-30T06:35:49.560ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c051be45-5469-449a-8c4f-f713cd6c1265", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
81872025-04-30T06:35:49.561ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
81882025-04-30T06:35:49.563ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8da852b8-64f1-4d5c-98b4-355e53c7955d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
81892025-04-30T06:35:49.563ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
81902025-04-30T06:35:49.564ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cff6c81-066d-47c5-a949-cb880310a6da", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
81912025-04-30T06:35:49.564ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
81922025-04-30T06:35:49.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f181476-9ed4-4bbd-b568-eae85e860a7d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
81932025-04-30T06:35:49.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
81942025-04-30T06:35:49.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
81952025-04-30T06:35:49.567ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
81962025-04-30T06:35:49.568ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
81972025-04-30T06:35:49.570ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "599cc56e-e956-459e-8b0f-a180dd8b678e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
81982025-04-30T06:35:49.570ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
81992025-04-30T06:35:49.571ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
82002025-04-30T06:35:49.571ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
82012025-04-30T06:35:49.572ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e6fe3f7-edbf-4a4c-ab67-dfe061010fe8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82022025-04-30T06:35:49.572ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
82032025-04-30T06:35:49.575ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "840eae1f-bba8-45d8-88c9-fccb240255c7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82042025-04-30T06:35:49.575ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
82052025-04-30T06:35:49.578ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b66ddfe5-23ed-496d-852d-40b2b019caad", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82062025-04-30T06:35:49.578ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
82072025-04-30T06:35:49.578ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
82082025-04-30T06:35:49.578ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82092025-04-30T06:35:49.580ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d884d37-890e-42ce-94c3-0f852b44f1b9", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82102025-04-30T06:35:49.581ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
82112025-04-30T06:35:49.582ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
82122025-04-30T06:35:49.582ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
82132025-04-30T06:35:49.583ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e757300a-3e1d-4b6b-a991-f90d9dc0afec", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82142025-04-30T06:35:49.583ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
82152025-04-30T06:35:49.584ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "844e094c-c74c-4b42-8c80-0894ad614343", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82162025-04-30T06:35:49.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
82172025-04-30T06:35:49.585ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83ca387e-bd7c-443a-a3c2-1870b62d0230", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82182025-04-30T06:35:49.586ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
82192025-04-30T06:35:49.586ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c057e3fd-624a-4fcc-811a-21edaaa7cf3f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82202025-04-30T06:35:49.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
82212025-04-30T06:35:49.588ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25d624d5-544c-45bd-a298-a5de924db5c7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82222025-04-30T06:35:49.588ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
82232025-04-30T06:35:49.590ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6f6fe3a-e420-4530-bb7d-7fffe28414d0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82242025-04-30T06:35:49.590ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
82252025-04-30T06:35:49.591ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d40572ba-1357-4b3c-b934-8f149f70b95a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82262025-04-30T06:35:49.591ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
82272025-04-30T06:35:49.593ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "21630c8c-4909-439e-83f0-87f837548a10", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82282025-04-30T06:35:49.593ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
82292025-04-30T06:35:49.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd3eebc3-6c71-4bea-ac4d-5a9391d8b082", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82302025-04-30T06:35:49.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
82312025-04-30T06:35:49.597ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3bbc8735-bed6-4bef-97cb-2a52dca5f03c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82322025-04-30T06:35:49.597ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
82332025-04-30T06:35:49.600ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d94f6a64-154c-4618-9cdd-6ee98cdb0857", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82342025-04-30T06:35:49.600ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
82352025-04-30T06:35:49.601ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71dbf75f-9b65-4085-9c05-84ecd0de9f99", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82362025-04-30T06:35:49.601ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
82372025-04-30T06:35:49.602ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a203d10-b5c5-4a49-9b5c-822285f03736", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82382025-04-30T06:35:49.603ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
82392025-04-30T06:35:49.605ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bae82a8b-67c5-4c58-a5a4-5f859ae97b93", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82402025-04-30T06:35:49.605ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
82412025-04-30T06:35:49.606ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82422025-04-30T06:35:49.606ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82432025-04-30T06:35:49.606ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82442025-04-30T06:35:49.606ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82452025-04-30T06:35:49.608ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82462025-04-30T06:35:49.609ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e3b44b6-d61b-42d7-9279-35a4be2786b6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82472025-04-30T06:35:49.610ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
82482025-04-30T06:35:49.610ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82492025-04-30T06:35:49.617ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
82502025-04-30T06:35:49.618ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82512025-04-30T06:35:49.618ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82522025-04-30T06:35:49.619ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82532025-04-30T06:35:49.620ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "595fda71-d185-4e52-b874-8d504d06ca50", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82542025-04-30T06:35:49.620ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
82552025-04-30T06:35:49.622ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
82562025-04-30T06:35:49.622ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
82572025-04-30T06:35:49.623ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
82582025-04-30T06:35:49.624ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a4bdb22-b685-4b0f-b9d7-145d02932eef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82592025-04-30T06:35:49.624ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
82602025-04-30T06:35:49.626ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b999b41-fa5f-422a-9d93-191711be7d06", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82612025-04-30T06:35:49.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
82622025-04-30T06:35:49.628ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "853fc2fa-ad45-49c9-9332-b08122128a04", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82632025-04-30T06:35:49.628ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
82642025-04-30T06:35:49.630ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3ea64a6-6de5-4b5f-80ed-ddcfe193a5d8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82652025-04-30T06:35:49.630ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
82662025-04-30T06:35:49.632ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13eb24a-e8d9-46cd-b589-607c420428a4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82672025-04-30T06:35:49.632ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
82682025-04-30T06:35:49.634ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55c10513-e0c3-42ce-b68f-e2eda098cd99", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82692025-04-30T06:35:49.635ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
82702025-04-30T06:35:49.637ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ba0cce6-d753-4e76-afab-a815c2b14ede", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82712025-04-30T06:35:49.637ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
82722025-04-30T06:35:49.639ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bee43b17-9d02-4830-81fd-db2606590a30", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82732025-04-30T06:35:49.639ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
82742025-04-30T06:35:49.641ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09680a98-4d3f-45da-b6f8-5f5075369374", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82752025-04-30T06:35:49.641ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
82762025-04-30T06:35:49.642ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "50ea4820-441c-45b3-a89e-a5b1f9ce5269", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
82772025-04-30T06:35:49.642ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
82782025-04-30T06:35:49.642ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82792025-04-30T06:35:49.643ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82802025-04-30T06:35:49.643ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82812025-04-30T06:35:49.644ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82822025-04-30T06:35:49.646ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82832025-04-30T06:35:49.649ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82842025-04-30T06:35:49.650ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
82852025-04-30T06:35:49.650ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82862025-04-30T06:35:49.650ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82872025-04-30T06:35:49.650ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82882025-04-30T06:35:49.650ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82892025-04-30T06:35:49.652ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
82902025-04-30T06:35:49.656ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
82912025-04-30T06:35:49.656ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
82922025-04-30T06:35:49.656ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
82932025-04-30T06:35:49.657ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
82942025-04-30T06:35:49.660ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
82952025-04-30T06:35:49.721ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
82962025-04-30T06:35:49.721ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
82972025-04-30T06:35:49.722ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
82982025-04-30T06:35:49.722ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
82992025-04-30T06:35:49.722ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83002025-04-30T06:35:49.723ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83012025-04-30T06:35:49.724ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83022025-04-30T06:35:49.724ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:46704 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
83032025-04-30T06:35:49.726ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
83042025-04-30T06:35:49.726ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:46704 req_id = e47af23c-2170-42ab-bb45-468878acc374 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83052025-04-30T06:35:49.727ZINFOtest_metrics (dropshot): request completed latency_us = 543 local_addr = [::1]:56237 method = GET remote_addr = [::1]:46704 req_id = e47af23c-2170-42ab-bb45-468878acc374 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
83062025-04-30T06:35:49.727ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "e47af23c-2170-42ab-bb45-468878acc374", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83072025-04-30T06:35:49.731ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83082025-04-30T06:35:49.731ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
83092025-04-30T06:35:49.731ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
83102025-04-30T06:35:49.732ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83112025-04-30T06:35:49.732ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83122025-04-30T06:35:49.732ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83132025-04-30T06:35:49.732ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83142025-04-30T06:35:49.733ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83152025-04-30T06:35:49.733ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83162025-04-30T06:35:49.738ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83172025-04-30T06:35:49.738ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83182025-04-30T06:35:49.738ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83192025-04-30T06:35:49.739ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83202025-04-30T06:35:49.741ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83212025-04-30T06:35:49.746ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83222025-04-30T06:35:49.747ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83232025-04-30T06:35:49.747ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83242025-04-30T06:35:49.748ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83252025-04-30T06:35:49.749ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83262025-04-30T06:35:49.750ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83272025-04-30T06:35:49.751ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83282025-04-30T06:35:49.790ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
83292025-04-30T06:35:49.790ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
83302025-04-30T06:35:49.790ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 439.76416ms iteration = 26 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
83312025-04-30T06:35:49.791ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 27 reason = Signaled
83322025-04-30T06:35:49.791ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
83332025-04-30T06:35:49.791ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83342025-04-30T06:35:49.791ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83352025-04-30T06:35:49.791ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83362025-04-30T06:35:49.792ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83372025-04-30T06:35:49.800ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83382025-04-30T06:35:49.808ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
83392025-04-30T06:35:49.808ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
83402025-04-30T06:35:49.808ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
83412025-04-30T06:35:49.808ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
83422025-04-30T06:35:49.827ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
83432025-04-30T06:35:49.827ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
83442025-04-30T06:35:49.828ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6c e1 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 l............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52443 req_id = 2640c7b8-5c7e-4e64-9a9d-da3bdae1478f
83452025-04-30T06:35:49.829ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 27873,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52443 req_id = 2640c7b8-5c7e-4e64-9a9d-da3bdae1478f
83462025-04-30T06:35:49.829ZDEBGtest_metrics (store): query key key = _mgs._tcp
83472025-04-30T06:35:49.829ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
83482025-04-30T06:35:49.829ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52443 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2640c7b8-5c7e-4e64-9a9d-da3bdae1478f
83492025-04-30T06:35:49.830ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
83502025-04-30T06:35:49.831ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3814, tv_nsec: 630142331 } })
83512025-04-30T06:35:49.831ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
83522025-04-30T06:35:49.832ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ed 3c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .<..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59115 req_id = 38a765ff-192d-4107-9046-7a540023af5a
83532025-04-30T06:35:49.832ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60732,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59115 req_id = 38a765ff-192d-4107-9046-7a540023af5a
83542025-04-30T06:35:49.833ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
83552025-04-30T06:35:49.833ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59115 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 38a765ff-192d-4107-9046-7a540023af5a
83562025-04-30T06:35:49.970ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
83572025-04-30T06:35:49.970ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
83582025-04-30T06:35:49.970ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
83592025-04-30T06:35:49.970ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83602025-04-30T06:35:49.970ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83612025-04-30T06:35:49.973ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83622025-04-30T06:35:49.974ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: e6 94 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:61277 req_id = 64c652da-9538-473e-acc5-3862f55a0c04
83632025-04-30T06:35:49.974ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59028,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61277 req_id = 64c652da-9538-473e-acc5-3862f55a0c04
83642025-04-30T06:35:49.974ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
83652025-04-30T06:35:49.974ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 59028, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:61277 req_id = 64c652da-9538-473e-acc5-3862f55a0c04
83662025-04-30T06:35:49.974ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
83672025-04-30T06:35:49.974ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
83682025-04-30T06:35:49.974ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
83692025-04-30T06:35:49.976ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
83702025-04-30T06:35:49.990ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
83712025-04-30T06:35:49.990ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83722025-04-30T06:35:49.990ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83732025-04-30T06:35:49.990ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83742025-04-30T06:35:49.994ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
83752025-04-30T06:35:49.994ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
83762025-04-30T06:35:49.995ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
83772025-04-30T06:35:49.995ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9a45d78-35b8-4d8e-82db-8ced43e65d3d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83782025-04-30T06:35:49.996ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
83792025-04-30T06:35:50.002ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a889049b-7bcc-456c-94c7-74b62b5a27fd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83802025-04-30T06:35:50.002ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
83812025-04-30T06:35:50.013ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "addb4db2-e96c-4f48-8057-5757b7818031", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83822025-04-30T06:35:50.014ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
83832025-04-30T06:35:50.035ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4e98c58-a0df-4464-9e1c-049d361bdca9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83842025-04-30T06:35:50.036ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
83852025-04-30T06:35:50.042ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "93141550-fc4e-4f3b-a4cc-e72b39d57430", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83862025-04-30T06:35:50.043ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
83872025-04-30T06:35:50.049ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31b80b72-b279-4b32-a1c1-28b98cee2e4d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83882025-04-30T06:35:50.049ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
83892025-04-30T06:35:50.054ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
83902025-04-30T06:35:50.059ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "070b3e09-927a-4902-9ea4-da7cdf90efa5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83912025-04-30T06:35:50.059ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
83922025-04-30T06:35:50.079ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "079f1ba3-664a-4be9-a51b-11fec0cf5cf0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83932025-04-30T06:35:50.080ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
83942025-04-30T06:35:50.082ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b6a760b-138e-4d33-8812-5c1046254a4a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83952025-04-30T06:35:50.087ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
83962025-04-30T06:35:50.102ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dff55b34-e586-4e92-bfa1-68e959129961", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83972025-04-30T06:35:50.102ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
83982025-04-30T06:35:50.103ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8aa4995f-381a-4d4a-b898-45dea770d4a7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
83992025-04-30T06:35:50.103ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
84002025-04-30T06:35:50.104ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "331a7e32-ce45-427b-a741-2c58c0dc2755", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84012025-04-30T06:35:50.104ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
84022025-04-30T06:35:50.105ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84032025-04-30T06:35:50.105ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84042025-04-30T06:35:50.105ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84052025-04-30T06:35:50.105ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84062025-04-30T06:35:50.107ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84072025-04-30T06:35:50.129ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5818198-8875-469c-baf8-55f6a15c025f", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84082025-04-30T06:35:50.130ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
84092025-04-30T06:35:50.130ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84102025-04-30T06:35:50.154ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84112025-04-30T06:35:50.154ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84122025-04-30T06:35:50.154ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84132025-04-30T06:35:50.155ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84142025-04-30T06:35:50.204ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84152025-04-30T06:35:50.204ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84162025-04-30T06:35:50.205ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84172025-04-30T06:35:50.209ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "745e38c5-bfc8-4e52-9d80-98ab992fae9f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84182025-04-30T06:35:50.210ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
84192025-04-30T06:35:50.213ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "13be18ae-5fe2-4030-ad0c-a0029ea5cc13", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84202025-04-30T06:35:50.213ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
84212025-04-30T06:35:50.226ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a2d006d-fccc-459d-b4b0-96d44ada07bf", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84222025-04-30T06:35:50.226ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
84232025-04-30T06:35:50.228ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d2ebe02-a3c1-467a-979c-cd20b7e34d69", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84242025-04-30T06:35:50.228ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
84252025-04-30T06:35:50.231ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a290c524-b6a1-4508-aa1f-361c9b50812d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84262025-04-30T06:35:50.231ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
84272025-04-30T06:35:50.233ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43dbde91-08b7-4f2c-9625-3c8b1e76014c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84282025-04-30T06:35:50.233ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
84292025-04-30T06:35:50.235ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf79d746-e2c0-4d51-a84c-6c2c49d92ecc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84302025-04-30T06:35:50.235ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
84312025-04-30T06:35:50.236ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6921bb2e-5a73-4375-806f-bf1b1ce04373", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84322025-04-30T06:35:50.237ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
84332025-04-30T06:35:50.238ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d7163d6-e479-4320-8760-f4e7784b5a0e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84342025-04-30T06:35:50.238ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
84352025-04-30T06:35:50.250ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6eb97da1-d975-4023-8351-64d1a5f86796", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84362025-04-30T06:35:50.250ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
84372025-04-30T06:35:50.254ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
84382025-04-30T06:35:50.254ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9582e0a-6096-465d-8f7f-ca1fef927301", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84392025-04-30T06:35:50.254ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
84402025-04-30T06:35:50.258ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f44d77e-f5b0-4698-9351-233afc3e6590", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84412025-04-30T06:35:50.258ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
84422025-04-30T06:35:50.274ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "96a11e91-50a9-4378-8231-e7d59770e1dc", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84432025-04-30T06:35:50.274ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
84442025-04-30T06:35:50.281ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "77744fbc-b0c2-42cb-995b-d6ad9aa8da27", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84452025-04-30T06:35:50.281ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
84462025-04-30T06:35:50.281ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99e15407-bf7d-4fd1-ae94-42a86ee0930c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84472025-04-30T06:35:50.281ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
84482025-04-30T06:35:50.284ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0e857077-e882-432c-912e-dc9bcb225d2c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84492025-04-30T06:35:50.284ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
84502025-04-30T06:35:50.290ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e4458b2-11a2-4dcd-b43e-0dd2ddfda631", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84512025-04-30T06:35:50.290ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
84522025-04-30T06:35:50.291ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "743d5c60-d509-4126-b1e0-4ed9ea66769a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84532025-04-30T06:35:50.291ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
84542025-04-30T06:35:50.292ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "129a9e71-0ec4-4fb9-b7e1-c08fe3865ca5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84552025-04-30T06:35:50.292ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
84562025-04-30T06:35:50.292ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bef2a05-2ad0-471a-9503-8c728d6b1380", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84572025-04-30T06:35:50.293ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
84582025-04-30T06:35:50.293ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ec4059e-e779-43c9-b12f-4852cbefc2f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84592025-04-30T06:35:50.294ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
84602025-04-30T06:35:50.295ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ec55294-a4db-40cd-a370-5945e2d4dd19", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84612025-04-30T06:35:50.295ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
84622025-04-30T06:35:50.296ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6d00647-7e3a-4ad8-85fa-b39a5c3c766a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84632025-04-30T06:35:50.296ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
84642025-04-30T06:35:50.298ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64fdcc31-d253-48a3-8287-cc6cbe984b7e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84652025-04-30T06:35:50.298ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
84662025-04-30T06:35:50.299ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "079aa2ed-459c-4d11-be49-ef111851c6d1", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84672025-04-30T06:35:50.299ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
84682025-04-30T06:35:50.301ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "be2fab67-6ee0-4e34-b2b4-63cb1bff6b8e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84692025-04-30T06:35:50.301ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
84702025-04-30T06:35:50.302ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b85c83fb-7b56-40aa-9268-0348eb7b6162", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84712025-04-30T06:35:50.302ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
84722025-04-30T06:35:50.304ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a3b743e-66a6-4d29-9d9f-a7cc6bdd2303", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84732025-04-30T06:35:50.304ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
84742025-04-30T06:35:50.305ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
84752025-04-30T06:35:50.305ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84762025-04-30T06:35:50.305ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84772025-04-30T06:35:50.305ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84782025-04-30T06:35:50.306ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84792025-04-30T06:35:50.306ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "09ee7ffd-2aba-4e37-ae48-446b0e847e81", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84802025-04-30T06:35:50.306ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
84812025-04-30T06:35:50.307ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "25785716-c07e-4553-b1eb-1e21f83b3a47", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84822025-04-30T06:35:50.308ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
84832025-04-30T06:35:50.308ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7197a5ef-8a04-4942-a701-a52d33b05483", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84842025-04-30T06:35:50.309ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
84852025-04-30T06:35:50.309ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
84862025-04-30T06:35:50.315ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
84872025-04-30T06:35:50.315ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84882025-04-30T06:35:50.315ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84892025-04-30T06:35:50.316ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84902025-04-30T06:35:50.319ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c325cee9-f2bb-4ea1-9209-560f340aff78", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
84912025-04-30T06:35:50.319ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
84922025-04-30T06:35:50.319ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
84932025-04-30T06:35:50.319ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
84942025-04-30T06:35:50.319ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
84952025-04-30T06:35:50.320ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
84962025-04-30T06:35:50.320ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
84972025-04-30T06:35:50.320ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
84982025-04-30T06:35:50.321ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
84992025-04-30T06:35:50.323ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85002025-04-30T06:35:50.328ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85012025-04-30T06:35:50.328ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
85022025-04-30T06:35:50.328ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85032025-04-30T06:35:50.328ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85042025-04-30T06:35:50.328ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85052025-04-30T06:35:50.329ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85062025-04-30T06:35:50.332ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85072025-04-30T06:35:50.338ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
85082025-04-30T06:35:50.338ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85092025-04-30T06:35:50.339ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85102025-04-30T06:35:50.339ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85112025-04-30T06:35:50.402ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
85122025-04-30T06:35:50.402ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
85132025-04-30T06:35:50.404ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:48913 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
85142025-04-30T06:35:50.408ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:48913 req_id = 20c0180a-8e4c-436c-b15d-97a2a1e26a8b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85152025-04-30T06:35:50.409ZINFOtest_metrics (dropshot): request completed latency_us = 1263 local_addr = [::1]:56237 method = GET remote_addr = [::1]:48913 req_id = 20c0180a-8e4c-436c-b15d-97a2a1e26a8b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
85162025-04-30T06:35:50.417ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "20c0180a-8e4c-436c-b15d-97a2a1e26a8b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:49 GMT"} })
85172025-04-30T06:35:50.421ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85182025-04-30T06:35:50.421ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
85192025-04-30T06:35:50.421ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
85202025-04-30T06:35:50.423ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
85212025-04-30T06:35:50.423ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85222025-04-30T06:35:50.423ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85232025-04-30T06:35:50.423ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85242025-04-30T06:35:50.424ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85252025-04-30T06:35:50.430ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85262025-04-30T06:35:50.434ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
85272025-04-30T06:35:50.434ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85282025-04-30T06:35:50.435ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85292025-04-30T06:35:50.435ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85302025-04-30T06:35:50.479ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
85312025-04-30T06:35:50.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85322025-04-30T06:35:50.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85332025-04-30T06:35:50.518ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85342025-04-30T06:35:50.519ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85352025-04-30T06:35:50.519ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85362025-04-30T06:35:50.519ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
85372025-04-30T06:35:50.519ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
85382025-04-30T06:35:50.520ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
85392025-04-30T06:35:50.520ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 729.048081ms iteration = 27 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
85402025-04-30T06:35:50.520ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 28 reason = Signaled
85412025-04-30T06:35:50.520ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
85422025-04-30T06:35:50.520ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85432025-04-30T06:35:50.520ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85442025-04-30T06:35:50.520ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85452025-04-30T06:35:50.529ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85462025-04-30T06:35:50.530ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
85472025-04-30T06:35:50.530ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
85482025-04-30T06:35:50.530ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85492025-04-30T06:35:50.530ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
85502025-04-30T06:35:50.530ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85512025-04-30T06:35:50.532ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85522025-04-30T06:35:50.539ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
85532025-04-30T06:35:50.539ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
85542025-04-30T06:35:50.539ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
85552025-04-30T06:35:50.539ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
85562025-04-30T06:35:50.556ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
85572025-04-30T06:35:50.564ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
85582025-04-30T06:35:50.564ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85592025-04-30T06:35:50.564ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
85602025-04-30T06:35:50.564ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85612025-04-30T06:35:50.565ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
85622025-04-30T06:35:50.565ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
85632025-04-30T06:35:50.573ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
85642025-04-30T06:35:50.573ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
85652025-04-30T06:35:50.574ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
85662025-04-30T06:35:50.574ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85672025-04-30T06:35:50.575ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85682025-04-30T06:35:50.575ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
85692025-04-30T06:35:50.576ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
85702025-04-30T06:35:50.577ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 8c 07 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55003 req_id = 15b2a9cd-bc59-42d6-aca4-099fc0bd1005
85712025-04-30T06:35:50.577ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 35847,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55003 req_id = 15b2a9cd-bc59-42d6-aca4-099fc0bd1005
85722025-04-30T06:35:50.577ZDEBGtest_metrics (store): query key key = _mgs._tcp
85732025-04-30T06:35:50.577ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
85742025-04-30T06:35:50.577ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55003 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 15b2a9cd-bc59-42d6-aca4-099fc0bd1005
85752025-04-30T06:35:50.578ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3815, tv_nsec: 377620012 } })
85762025-04-30T06:35:50.578ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
85772025-04-30T06:35:50.579ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 0b e7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56347 req_id = a93dd5e0-058e-4602-8a47-266b088a0c11
85782025-04-30T06:35:50.579ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3047,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56347 req_id = a93dd5e0-058e-4602-8a47-266b088a0c11
85792025-04-30T06:35:50.580ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
85802025-04-30T06:35:50.580ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56347 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = a93dd5e0-058e-4602-8a47-266b088a0c11
85812025-04-30T06:35:50.581ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
85822025-04-30T06:35:50.582ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
85832025-04-30T06:35:50.633ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
85842025-04-30T06:35:50.635ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: ad c5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54783 req_id = b5cc7b8a-1c8c-4637-8aec-2268a4a9e9a8
85852025-04-30T06:35:50.635ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44485,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54783 req_id = b5cc7b8a-1c8c-4637-8aec-2268a4a9e9a8
85862025-04-30T06:35:50.635ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
85872025-04-30T06:35:50.635ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 44485, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54783 req_id = b5cc7b8a-1c8c-4637-8aec-2268a4a9e9a8
85882025-04-30T06:35:50.637ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
85892025-04-30T06:35:50.637ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
85902025-04-30T06:35:50.637ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
85912025-04-30T06:35:50.639ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
85922025-04-30T06:35:50.639ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
85932025-04-30T06:35:50.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "6697390a-dba6-43ab-8150-f64fde85e050", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
85942025-04-30T06:35:50.647ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
85952025-04-30T06:35:50.656ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79c85620-1a3b-4571-9415-cbd5ec3b0265", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
85962025-04-30T06:35:50.657ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
85972025-04-30T06:35:50.664ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e021b2b-431d-4983-bdb4-8168fd3cabc2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
85982025-04-30T06:35:50.665ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
85992025-04-30T06:35:50.669ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
86002025-04-30T06:35:50.670ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9831e613-c4a3-4323-9e65-667cf0ab5815", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86012025-04-30T06:35:50.670ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
86022025-04-30T06:35:50.676ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
86032025-04-30T06:35:50.677ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "185aff69-7c04-469c-a6e0-4c574f31bd89", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86042025-04-30T06:35:50.677ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
86052025-04-30T06:35:50.679ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3757b29c-b840-4e46-bdf3-efc15862efa6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86062025-04-30T06:35:50.679ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
86072025-04-30T06:35:50.681ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea722f15-bf8f-444f-8808-0ac4de035c3b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86082025-04-30T06:35:50.681ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
86092025-04-30T06:35:50.684ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "656bb2aa-d5c9-40bf-a007-a1f09202f61b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86102025-04-30T06:35:50.684ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
86112025-04-30T06:35:50.686ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6466d2a1-fb71-449a-b04b-552c653ae875", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86122025-04-30T06:35:50.686ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
86132025-04-30T06:35:50.687ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66c5e994-0d92-4d79-9481-ec558b10d915", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86142025-04-30T06:35:50.687ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
86152025-04-30T06:35:50.688ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f38876-da47-4c00-8a9a-49e0ecbbb700", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86162025-04-30T06:35:50.688ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
86172025-04-30T06:35:50.689ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2a5e472-4a1f-45c7-9f93-c684cc1aaefe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86182025-04-30T06:35:50.690ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
86192025-04-30T06:35:50.694ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ad5c924-5f4e-4a59-aaf6-b2b3f3a40288", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86202025-04-30T06:35:50.694ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
86212025-04-30T06:35:50.698ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd350dfa-6b74-4175-9159-08803c9df326", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86222025-04-30T06:35:50.698ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
86232025-04-30T06:35:50.700ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "314eaa0a-564b-49e3-a3dd-630a16e2140f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86242025-04-30T06:35:50.700ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
86252025-04-30T06:35:50.702ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7c758f8-1830-429c-8062-f3df9cb5c07d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86262025-04-30T06:35:50.702ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
86272025-04-30T06:35:50.704ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "740c31d2-c0bc-4363-a4f5-c3658171d962", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86282025-04-30T06:35:50.704ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
86292025-04-30T06:35:50.706ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1caf9ba7-baca-410d-abf8-854604930efc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86302025-04-30T06:35:50.706ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
86312025-04-30T06:35:50.709ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d7236d-27a2-4942-b2b6-6024e2abdced", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86322025-04-30T06:35:50.709ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
86332025-04-30T06:35:50.710ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "208ff3ed-9c38-4872-81fe-af34811f0f5b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86342025-04-30T06:35:50.710ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
86352025-04-30T06:35:50.711ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "235918b4-7b5b-4196-aac7-28372394fbd8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86362025-04-30T06:35:50.711ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
86372025-04-30T06:35:50.712ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b9f4e63-6a30-4e2a-ad79-df38930e9acc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86382025-04-30T06:35:50.712ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
86392025-04-30T06:35:50.713ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "169cafc8-e158-4174-8066-79536cc66028", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86402025-04-30T06:35:50.713ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
86412025-04-30T06:35:50.715ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d82547e-d1f9-4236-946b-7a64b7cd8a55", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86422025-04-30T06:35:50.716ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
86432025-04-30T06:35:50.718ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e869cb9f-84cf-42b0-af12-323add5433c2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86442025-04-30T06:35:50.719ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
86452025-04-30T06:35:50.720ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
86462025-04-30T06:35:50.720ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
86472025-04-30T06:35:50.720ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86482025-04-30T06:35:50.720ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86492025-04-30T06:35:50.721ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86502025-04-30T06:35:50.723ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
86512025-04-30T06:35:50.729ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
86522025-04-30T06:35:50.729ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86532025-04-30T06:35:50.729ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86542025-04-30T06:35:50.729ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86552025-04-30T06:35:50.731ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35376ea2-0490-496a-a8fc-d0c3cd1a9b96", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86562025-04-30T06:35:50.732ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
86572025-04-30T06:35:50.733ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
86582025-04-30T06:35:50.733ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
86592025-04-30T06:35:50.733ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
86602025-04-30T06:35:50.736ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "01b80e02-050a-441f-841a-82a1eb369c20", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86612025-04-30T06:35:50.736ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
86622025-04-30T06:35:50.744ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42f0f859-c344-4432-9dba-7e6f6b6b2f0b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86632025-04-30T06:35:50.744ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
86642025-04-30T06:35:50.747ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1debcd9f-964b-4d06-a4a3-4f40b16e97de", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86652025-04-30T06:35:50.747ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
86662025-04-30T06:35:50.753ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3fab22b-641a-46db-ad28-26e7379e0ee5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86672025-04-30T06:35:50.753ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
86682025-04-30T06:35:50.756ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1fdd597f-cd68-45d3-b910-63a8305ac282", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86692025-04-30T06:35:50.756ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
86702025-04-30T06:35:50.757ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd31bcf-27bb-4315-a47d-361dda7cdabd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86712025-04-30T06:35:50.758ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
86722025-04-30T06:35:50.759ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8a6645f3-21c8-48ab-b36e-27c6e6b1e766", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86732025-04-30T06:35:50.759ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
86742025-04-30T06:35:50.760ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3e367a5a-4a32-489d-b4ef-25ea802f46fe", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86752025-04-30T06:35:50.760ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
86762025-04-30T06:35:50.762ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "55344cea-62a1-4813-816b-d69906b9e6d7", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86772025-04-30T06:35:50.762ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
86782025-04-30T06:35:50.764ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
86792025-04-30T06:35:50.766ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c4204739-865f-4c02-b322-6d864ae06ee9", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86802025-04-30T06:35:50.766ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
86812025-04-30T06:35:50.768ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c006c08-5558-463e-a390-5d068e4ea189", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86822025-04-30T06:35:50.768ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
86832025-04-30T06:35:50.770ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdcdfe30-c0da-4a7a-a2b9-1166f40c8771", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86842025-04-30T06:35:50.771ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
86852025-04-30T06:35:50.773ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e611cacf-e239-42dd-9e07-d330e1be1eac", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86862025-04-30T06:35:50.774ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
86872025-04-30T06:35:50.776ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f5fcd77-c5c9-44d8-8cac-7660a86cd783", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86882025-04-30T06:35:50.776ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
86892025-04-30T06:35:50.778ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe34b87b-8d72-4bc8-b837-91b5dc9cd0d4", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86902025-04-30T06:35:50.778ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
86912025-04-30T06:35:50.779ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "28e28abd-d260-442e-8ca8-1eb26974a1b7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86922025-04-30T06:35:50.779ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
86932025-04-30T06:35:50.781ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4d59e90-0ff2-402a-9a60-9900c66528a4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86942025-04-30T06:35:50.781ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
86952025-04-30T06:35:50.782ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b098fd7-5d5f-408c-a82c-f26e66f4fdd3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86962025-04-30T06:35:50.783ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
86972025-04-30T06:35:50.784ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d5b873a-660c-4c32-94af-a8a6802758e8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
86982025-04-30T06:35:50.784ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
86992025-04-30T06:35:50.784ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87002025-04-30T06:35:50.784ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87012025-04-30T06:35:50.784ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87022025-04-30T06:35:50.785ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87032025-04-30T06:35:50.789ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87042025-04-30T06:35:50.792ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87052025-04-30T06:35:50.792ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
87062025-04-30T06:35:50.792ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87072025-04-30T06:35:50.792ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87082025-04-30T06:35:50.792ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87092025-04-30T06:35:50.792ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87102025-04-30T06:35:50.796ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87112025-04-30T06:35:50.799ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
87122025-04-30T06:35:50.800ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87132025-04-30T06:35:50.800ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87142025-04-30T06:35:50.801ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87152025-04-30T06:35:50.849ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
87162025-04-30T06:35:50.849ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
87172025-04-30T06:35:50.849ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87182025-04-30T06:35:50.849ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87192025-04-30T06:35:50.849ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87202025-04-30T06:35:50.849ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87212025-04-30T06:35:50.850ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87222025-04-30T06:35:50.850ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:54899 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
87232025-04-30T06:35:50.851ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:54899 req_id = 67446695-40a5-4530-aee6-b46a59c55d37 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
87242025-04-30T06:35:50.852ZINFOtest_metrics (dropshot): request completed latency_us = 572 local_addr = [::1]:56237 method = GET remote_addr = [::1]:54899 req_id = 67446695-40a5-4530-aee6-b46a59c55d37 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
87252025-04-30T06:35:50.852ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "67446695-40a5-4530-aee6-b46a59c55d37", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
87262025-04-30T06:35:50.854ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
87272025-04-30T06:35:50.854ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
87282025-04-30T06:35:50.854ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
87292025-04-30T06:35:50.855ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
87302025-04-30T06:35:50.856ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87312025-04-30T06:35:50.856ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87322025-04-30T06:35:50.856ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87332025-04-30T06:35:50.856ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87342025-04-30T06:35:50.857ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87352025-04-30T06:35:50.862ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87362025-04-30T06:35:50.878ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87372025-04-30T06:35:50.878ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87382025-04-30T06:35:50.879ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87392025-04-30T06:35:50.884ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87402025-04-30T06:35:50.889ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87412025-04-30T06:35:50.889ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87422025-04-30T06:35:50.889ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87432025-04-30T06:35:50.890ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87442025-04-30T06:35:50.894ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87452025-04-30T06:35:50.900ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87462025-04-30T06:35:50.901ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87472025-04-30T06:35:50.935ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
87482025-04-30T06:35:50.936ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
87492025-04-30T06:35:50.936ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 415.911928ms iteration = 28 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
87502025-04-30T06:35:50.936ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 29 reason = Signaled
87512025-04-30T06:35:50.936ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
87522025-04-30T06:35:50.936ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87532025-04-30T06:35:50.936ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87542025-04-30T06:35:50.936ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87552025-04-30T06:35:50.937ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87562025-04-30T06:35:50.941ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87572025-04-30T06:35:50.946ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
87582025-04-30T06:35:50.946ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
87592025-04-30T06:35:50.947ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
87602025-04-30T06:35:50.948ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
87612025-04-30T06:35:50.953ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
87622025-04-30T06:35:50.955ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
87632025-04-30T06:35:50.957ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
87642025-04-30T06:35:50.958ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 18 3b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .;..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65492 req_id = 293e4e96-2db5-44c9-ad05-9939fc8ddfb8
87652025-04-30T06:35:50.958ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6203,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65492 req_id = 293e4e96-2db5-44c9-ad05-9939fc8ddfb8
87662025-04-30T06:35:50.959ZDEBGtest_metrics (store): query key key = _mgs._tcp
87672025-04-30T06:35:50.960ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87682025-04-30T06:35:50.960ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65492 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 293e4e96-2db5-44c9-ad05-9939fc8ddfb8
87692025-04-30T06:35:50.960ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3815, tv_nsec: 759657878 } })
87702025-04-30T06:35:50.960ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
87712025-04-30T06:35:50.961ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 29 14 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e )...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:52086 req_id = c3c96037-487b-4cc5-a6ca-26e4f0ae4a34
87722025-04-30T06:35:50.961ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10516,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52086 req_id = c3c96037-487b-4cc5-a6ca-26e4f0ae4a34
87732025-04-30T06:35:50.963ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
87742025-04-30T06:35:50.963ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:52086 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c3c96037-487b-4cc5-a6ca-26e4f0ae4a34
87752025-04-30T06:35:51.012ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
87762025-04-30T06:35:51.013ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
87772025-04-30T06:35:51.013ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
87782025-04-30T06:35:51.013ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87792025-04-30T06:35:51.013ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87802025-04-30T06:35:51.014ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87812025-04-30T06:35:51.014ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: a5 15 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54184 req_id = 0eccc548-06e8-4d15-a54d-b6e5f3588218
87822025-04-30T06:35:51.014ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42261,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54184 req_id = 0eccc548-06e8-4d15-a54d-b6e5f3588218
87832025-04-30T06:35:51.015ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
87842025-04-30T06:35:51.015ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 42261, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54184 req_id = 0eccc548-06e8-4d15-a54d-b6e5f3588218
87852025-04-30T06:35:51.015ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
87862025-04-30T06:35:51.015ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
87872025-04-30T06:35:51.015ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
87882025-04-30T06:35:51.017ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
87892025-04-30T06:35:51.022ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
87902025-04-30T06:35:51.022ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87912025-04-30T06:35:51.022ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87922025-04-30T06:35:51.023ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87932025-04-30T06:35:51.025ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "43c9f121-1621-4ac1-b059-d8c47b54919c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
87942025-04-30T06:35:51.027ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
87952025-04-30T06:35:51.027ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
87962025-04-30T06:35:51.027ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
87972025-04-30T06:35:51.028ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
87982025-04-30T06:35:51.031ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a74aaf7f-851b-4096-9a48-0b76aa7b6bf1", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
87992025-04-30T06:35:51.031ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
88002025-04-30T06:35:51.036ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de6e8b1b-7d09-4e91-a231-2eeb9f8c11fb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88012025-04-30T06:35:51.036ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
88022025-04-30T06:35:51.039ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "768a4dd7-3bdc-49d8-b319-de7d7169b662", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88032025-04-30T06:35:51.039ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
88042025-04-30T06:35:51.043ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba5c06a4-94fc-4073-baa7-f7281d0c2a04", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88052025-04-30T06:35:51.043ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
88062025-04-30T06:35:51.046ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f54b7fba-7eca-4544-85b9-bb530e3fb32c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88072025-04-30T06:35:51.046ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
88082025-04-30T06:35:51.047ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "215be729-1afd-40fe-915f-be0e2252ac23", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88092025-04-30T06:35:51.047ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
88102025-04-30T06:35:51.050ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "39ff659c-67cc-4024-a09b-c910d4bef3ac", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88112025-04-30T06:35:51.050ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
88122025-04-30T06:35:51.051ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52a875f6-3ec0-43b6-8ce4-6149006a55d5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88132025-04-30T06:35:51.051ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
88142025-04-30T06:35:51.052ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c04d2bac-2947-40b6-8aca-b033a5c27723", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88152025-04-30T06:35:51.053ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
88162025-04-30T06:35:51.054ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8d810cb5-8a46-48f5-9f83-c7c740be5d7d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88172025-04-30T06:35:51.054ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
88182025-04-30T06:35:51.055ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "559524fd-96d9-42e7-bb5b-badd3c1ba316", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88192025-04-30T06:35:51.055ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
88202025-04-30T06:35:51.059ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6c8b627-a118-43f0-b2a2-7f0a532059c0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88212025-04-30T06:35:51.059ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
88222025-04-30T06:35:51.063ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
88232025-04-30T06:35:51.068ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd614fad-1dfd-46ef-a511-91f9597d9e3d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88242025-04-30T06:35:51.068ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
88252025-04-30T06:35:51.071ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c5315f9c-4b14-494c-a136-ffc23d30265a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88262025-04-30T06:35:51.071ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
88272025-04-30T06:35:51.074ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1546ba26-0574-46b8-97b4-38ea2343f056", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88282025-04-30T06:35:51.074ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
88292025-04-30T06:35:51.078ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32987799-94f3-418c-831f-bf89cd7a54e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88302025-04-30T06:35:51.079ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
88312025-04-30T06:35:51.081ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bfac2531-27aa-47ff-8be9-f635cf6c6bc4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88322025-04-30T06:35:51.081ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
88332025-04-30T06:35:51.083ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d57921c-03a2-490f-8e43-814740c3e11b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88342025-04-30T06:35:51.083ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
88352025-04-30T06:35:51.084ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ecb79d9-cd8c-4163-8b7e-95897b658bcb", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88362025-04-30T06:35:51.084ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
88372025-04-30T06:35:51.085ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b5f4fea-c365-4f34-b45a-4bd010cb3380", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88382025-04-30T06:35:51.085ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
88392025-04-30T06:35:51.086ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61fba6ae-4aeb-4b98-b7e0-0b1f9d910b47", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88402025-04-30T06:35:51.086ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
88412025-04-30T06:35:51.088ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a61e3dd-cda5-456d-8c7f-bd9c3127ba05", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88422025-04-30T06:35:51.088ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
88432025-04-30T06:35:51.089ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "52a9cea6-d4d2-41ac-8848-4e7656f8bc40", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88442025-04-30T06:35:51.089ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
88452025-04-30T06:35:51.092ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5956fcae-1291-443b-8996-fa11a1aa8c6b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88462025-04-30T06:35:51.092ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
88472025-04-30T06:35:51.093ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d7bdc28e-d7c4-4814-b56d-e08442545753", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88482025-04-30T06:35:51.094ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
88492025-04-30T06:35:51.095ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f385d56f-893f-4786-90a3-b705ff729ab8", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88502025-04-30T06:35:51.095ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
88512025-04-30T06:35:51.097ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "173fb90e-6c00-4b71-a038-0898e6d4424c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88522025-04-30T06:35:51.097ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
88532025-04-30T06:35:51.099ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "39365e9f-5b5b-4c36-9d6b-e17a99e61fd2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88542025-04-30T06:35:51.100ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
88552025-04-30T06:35:51.105ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "93da9657-b49a-46d8-8ce5-eed0542356c3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88562025-04-30T06:35:51.105ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
88572025-04-30T06:35:51.107ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "047a377e-7a57-4908-be64-0bd6d4d339e3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88582025-04-30T06:35:51.107ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
88592025-04-30T06:35:51.109ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9874f5a8-6b9b-43da-86d0-f3b3437b3a3a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88602025-04-30T06:35:51.110ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
88612025-04-30T06:35:51.110ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f21b8e82-afbf-4dc7-a614-6a84a04df4d6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88622025-04-30T06:35:51.111ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
88632025-04-30T06:35:51.115ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
88642025-04-30T06:35:51.115ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
88652025-04-30T06:35:51.115ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88662025-04-30T06:35:51.115ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88672025-04-30T06:35:51.116ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88682025-04-30T06:35:51.117ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "30f599b1-0fd3-4000-bb04-1de630fb694d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88692025-04-30T06:35:51.117ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
88702025-04-30T06:35:51.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a55d2510-3d5a-4a5c-b046-39e81f5e8dfa", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88712025-04-30T06:35:51.122ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
88722025-04-30T06:35:51.122ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
88732025-04-30T06:35:51.131ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
88742025-04-30T06:35:51.131ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88752025-04-30T06:35:51.131ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88762025-04-30T06:35:51.131ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88772025-04-30T06:35:51.134ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
88782025-04-30T06:35:51.134ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
88792025-04-30T06:35:51.135ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
88802025-04-30T06:35:51.137ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf6a2492-f92e-4f53-8bbf-c67a6e89967c", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88812025-04-30T06:35:51.137ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
88822025-04-30T06:35:51.140ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "652d7bf7-2374-460a-811e-00e511246bd1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88832025-04-30T06:35:51.141ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
88842025-04-30T06:35:51.147ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "58de389d-895d-47dd-b23c-6a9743804174", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88852025-04-30T06:35:51.147ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
88862025-04-30T06:35:51.150ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3fada404-632d-4cd4-a9f5-d3d8ea01f03b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88872025-04-30T06:35:51.150ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
88882025-04-30T06:35:51.152ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "188019b9-aefc-447b-9bab-ab1fcfd47b60", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88892025-04-30T06:35:51.153ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
88902025-04-30T06:35:51.155ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bbc1d1b-68ce-4d50-9499-5105a0380a79", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88912025-04-30T06:35:51.155ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
88922025-04-30T06:35:51.156ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "111fce33-d727-4497-ab98-d59fda930103", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88932025-04-30T06:35:51.156ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
88942025-04-30T06:35:51.157ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "934c5226-8830-4080-8286-586e526695f7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88952025-04-30T06:35:51.157ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
88962025-04-30T06:35:51.159ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f532b99-ed36-480f-8235-69137d6b5093", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88972025-04-30T06:35:51.159ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
88982025-04-30T06:35:51.160ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "86f69fad-4ec1-40ab-bd2b-869c8a113a3e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
88992025-04-30T06:35:51.160ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89002025-04-30T06:35:51.160ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89012025-04-30T06:35:51.160ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89022025-04-30T06:35:51.160ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89032025-04-30T06:35:51.161ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89042025-04-30T06:35:51.163ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89052025-04-30T06:35:51.168ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89062025-04-30T06:35:51.168ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
89072025-04-30T06:35:51.168ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89082025-04-30T06:35:51.168ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89092025-04-30T06:35:51.168ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89102025-04-30T06:35:51.168ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89112025-04-30T06:35:51.173ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89122025-04-30T06:35:51.179ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
89132025-04-30T06:35:51.179ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89142025-04-30T06:35:51.179ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89152025-04-30T06:35:51.180ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89162025-04-30T06:35:51.181ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
89172025-04-30T06:35:51.243ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
89182025-04-30T06:35:51.243ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
89192025-04-30T06:35:51.244ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89202025-04-30T06:35:51.244ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89212025-04-30T06:35:51.244ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89222025-04-30T06:35:51.244ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
89232025-04-30T06:35:51.245ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89242025-04-30T06:35:51.245ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:63032 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
89252025-04-30T06:35:51.248ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:63032 req_id = 65b4e0d7-e300-4cd4-96c2-324d0c40a617 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
89262025-04-30T06:35:51.248ZINFOtest_metrics (dropshot): request completed latency_us = 551 local_addr = [::1]:56237 method = GET remote_addr = [::1]:63032 req_id = 65b4e0d7-e300-4cd4-96c2-324d0c40a617 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
89272025-04-30T06:35:51.248ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "65b4e0d7-e300-4cd4-96c2-324d0c40a617", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
89282025-04-30T06:35:51.251ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
89292025-04-30T06:35:51.251ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
89302025-04-30T06:35:51.251ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
89312025-04-30T06:35:51.254ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
89322025-04-30T06:35:51.254ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89332025-04-30T06:35:51.255ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89342025-04-30T06:35:51.255ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89352025-04-30T06:35:51.255ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89362025-04-30T06:35:51.256ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89372025-04-30T06:35:51.262ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
89382025-04-30T06:35:51.262ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89392025-04-30T06:35:51.262ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
89402025-04-30T06:35:51.262ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89412025-04-30T06:35:51.265ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89422025-04-30T06:35:51.270ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
89432025-04-30T06:35:51.270ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89442025-04-30T06:35:51.272ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89452025-04-30T06:35:51.272ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89462025-04-30T06:35:51.275ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89472025-04-30T06:35:51.275ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
89482025-04-30T06:35:51.277ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
89492025-04-30T06:35:51.296ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
89502025-04-30T06:35:51.297ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
89512025-04-30T06:35:51.298ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4aefa3e3-6684-4cac-9b6d-b196b56d96a0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
89522025-04-30T06:35:51.298ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4aefa3e3-6684-4cac-9b6d-b196b56d96a0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
89532025-04-30T06:35:51.303ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4aefa3e3-6684-4cac-9b6d-b196b56d96a0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
89542025-04-30T06:35:51.304ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4aefa3e3-6684-4cac-9b6d-b196b56d96a0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
89552025-04-30T06:35:51.308ZINFOtest_metrics (dropshot_internal): request completed latency_us = 10121 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4aefa3e3-6684-4cac-9b6d-b196b56d96a0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
89562025-04-30T06:35:51.308ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "4aefa3e3-6684-4cac-9b6d-b196b56d96a0", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
89572025-04-30T06:35:51.309ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
89582025-04-30T06:35:51.309ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4d128b9f-0ad2-424b-9a89-7b82d95360f6 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
89592025-04-30T06:35:51.310ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4d128b9f-0ad2-424b-9a89-7b82d95360f6 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
89602025-04-30T06:35:51.310ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4d128b9f-0ad2-424b-9a89-7b82d95360f6 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
89612025-04-30T06:35:51.311ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4d128b9f-0ad2-424b-9a89-7b82d95360f6 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
89622025-04-30T06:35:51.316ZINFOtest_metrics (dropshot_internal): request completed latency_us = 6663 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4d128b9f-0ad2-424b-9a89-7b82d95360f6 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
89632025-04-30T06:35:51.317ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d128b9f-0ad2-424b-9a89-7b82d95360f6", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
89642025-04-30T06:35:51.317ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
89652025-04-30T06:35:51.317ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
89662025-04-30T06:35:51.317ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
89672025-04-30T06:35:51.325ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
89682025-04-30T06:35:51.325ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
89692025-04-30T06:35:51.325ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 389.46818ms iteration = 29 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
89702025-04-30T06:35:51.325ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 30 reason = Signaled
89712025-04-30T06:35:51.325ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
89722025-04-30T06:35:51.325ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89732025-04-30T06:35:51.325ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89742025-04-30T06:35:51.325ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89752025-04-30T06:35:51.326ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89762025-04-30T06:35:51.330ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
89772025-04-30T06:35:51.332ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
89782025-04-30T06:35:51.336ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
89792025-04-30T06:35:51.337ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
89802025-04-30T06:35:51.337ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
89812025-04-30T06:35:51.337ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
89822025-04-30T06:35:51.342ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
89832025-04-30T06:35:51.342ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
89842025-04-30T06:35:51.346ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 25 c5 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 %............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60992 req_id = b3002897-9ce7-4f23-aada-2bac0855b63a
89852025-04-30T06:35:51.346ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 9669,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60992 req_id = b3002897-9ce7-4f23-aada-2bac0855b63a
89862025-04-30T06:35:51.346ZDEBGtest_metrics (store): query key key = _mgs._tcp
89872025-04-30T06:35:51.347ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89882025-04-30T06:35:51.347ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60992 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b3002897-9ce7-4f23-aada-2bac0855b63a
89892025-04-30T06:35:51.348ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3816, tv_nsec: 147268787 } })
89902025-04-30T06:35:51.348ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
89912025-04-30T06:35:51.351ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 40 1d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e @...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53521 req_id = b002e70b-0566-4ddc-9944-dde592308ef0
89922025-04-30T06:35:51.351ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16413,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53521 req_id = b002e70b-0566-4ddc-9944-dde592308ef0
89932025-04-30T06:35:51.352ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
89942025-04-30T06:35:51.352ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53521 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b002e70b-0566-4ddc-9944-dde592308ef0
89952025-04-30T06:35:51.404ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
89962025-04-30T06:35:51.409ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
89972025-04-30T06:35:51.409ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
89982025-04-30T06:35:51.409ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
89992025-04-30T06:35:51.409ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90002025-04-30T06:35:51.410ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90012025-04-30T06:35:51.412ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: c4 92 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59914 req_id = 31f5db94-7930-4b0b-9bf2-e293cf637757
90022025-04-30T06:35:51.412ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 50322,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59914 req_id = 31f5db94-7930-4b0b-9bf2-e293cf637757
90032025-04-30T06:35:51.412ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
90042025-04-30T06:35:51.412ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 50322, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59914 req_id = 31f5db94-7930-4b0b-9bf2-e293cf637757
90052025-04-30T06:35:51.414ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
90062025-04-30T06:35:51.414ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
90072025-04-30T06:35:51.414ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
90082025-04-30T06:35:51.415ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
90092025-04-30T06:35:51.420ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
90102025-04-30T06:35:51.420ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90112025-04-30T06:35:51.420ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90122025-04-30T06:35:51.420ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90132025-04-30T06:35:51.432ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90142025-04-30T06:35:51.432ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90152025-04-30T06:35:51.433ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90162025-04-30T06:35:51.443ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "80c9f695-eca9-4e58-8721-a183e27bdc8c", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90172025-04-30T06:35:51.444ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
90182025-04-30T06:35:51.446ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4393703c-82c7-4a14-be29-24826d2a471e", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90192025-04-30T06:35:51.447ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
90202025-04-30T06:35:51.449ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "12903c74-9fb8-4178-9419-21f379b37ca6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90212025-04-30T06:35:51.449ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
90222025-04-30T06:35:51.452ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aed910ba-0942-41f9-b64c-e2806253a916", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90232025-04-30T06:35:51.452ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
90242025-04-30T06:35:51.454ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "668f556f-2325-4435-860f-10a483561ca9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90252025-04-30T06:35:51.454ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
90262025-04-30T06:35:51.456ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "76c8ee1f-d34d-4e4f-984d-cad37d6c3542", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90272025-04-30T06:35:51.456ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
90282025-04-30T06:35:51.459ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c6e8304-b2f8-4bad-956a-dff10b14e936", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90292025-04-30T06:35:51.459ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
90302025-04-30T06:35:51.462ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "89ba67c9-9360-4c44-af18-9905cedd635f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90312025-04-30T06:35:51.463ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
90322025-04-30T06:35:51.464ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "11cea951-d7ce-422d-99ee-965c152f02dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90332025-04-30T06:35:51.465ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
90342025-04-30T06:35:51.467ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83b6f4aa-7293-4b09-b264-05bafba9dae0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90352025-04-30T06:35:51.467ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
90362025-04-30T06:35:51.474ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "625aac1f-0f1e-45a3-9482-a5651addfda9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90372025-04-30T06:35:51.474ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
90382025-04-30T06:35:51.475ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
90392025-04-30T06:35:51.476ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "448656a7-934c-4360-a58d-7a7ef3ce31ae", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90402025-04-30T06:35:51.476ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
90412025-04-30T06:35:51.478ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "babd9907-e634-4c60-a7ac-49108138bf2b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90422025-04-30T06:35:51.478ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
90432025-04-30T06:35:51.482ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b101e429-eac7-406e-9cc6-92e182a78c43", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90442025-04-30T06:35:51.482ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
90452025-04-30T06:35:51.485ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "67a057f0-ea5b-430a-878c-a9937e5f7d3a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90462025-04-30T06:35:51.485ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
90472025-04-30T06:35:51.488ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9fdb65c-e2ef-4bfe-bbee-22b14334a3e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90482025-04-30T06:35:51.488ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
90492025-04-30T06:35:51.491ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6ccda267-814d-4b9a-8dc4-6bf75b87785a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90502025-04-30T06:35:51.491ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
90512025-04-30T06:35:51.493ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c403472-a4cd-4736-9824-a2a923ec8493", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90522025-04-30T06:35:51.493ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
90532025-04-30T06:35:51.496ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "70007e80-c5ef-4daa-a7dd-126ad27d695b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90542025-04-30T06:35:51.496ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
90552025-04-30T06:35:51.497ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad17dc87-baae-4ef1-98c9-8b0d202c3fe4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90562025-04-30T06:35:51.497ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
90572025-04-30T06:35:51.498ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a89127cf-c95a-4b07-ad5f-4b30c73793e9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90582025-04-30T06:35:51.498ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
90592025-04-30T06:35:51.499ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9140f117-e749-4578-a4cb-5a7a11f11a66", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90602025-04-30T06:35:51.499ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
90612025-04-30T06:35:51.500ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f4db8bb-5829-44ee-a252-6dbcbb776850", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90622025-04-30T06:35:51.501ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
90632025-04-30T06:35:51.502ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e76dfc4-b733-4726-83f4-36d441b134c4", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90642025-04-30T06:35:51.502ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
90652025-04-30T06:35:51.504ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "635d0181-fe18-48f9-bad0-ccdeea8953fa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90662025-04-30T06:35:51.504ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
90672025-04-30T06:35:51.507ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e50edd6-ce93-4dce-b045-55a261b010d0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90682025-04-30T06:35:51.507ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
90692025-04-30T06:35:51.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c59fdac-75ca-47a6-8c5c-c0609117462d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90702025-04-30T06:35:51.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
90712025-04-30T06:35:51.511ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "04b592bb-0073-4370-b3ba-1507030c8722", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90722025-04-30T06:35:51.511ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
90732025-04-30T06:35:51.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90742025-04-30T06:35:51.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90752025-04-30T06:35:51.514ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90762025-04-30T06:35:51.515ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90772025-04-30T06:35:51.515ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90782025-04-30T06:35:51.515ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
90792025-04-30T06:35:51.516ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d1e5b96-1a26-428e-a797-ea85aed98c94", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90802025-04-30T06:35:51.516ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
90812025-04-30T06:35:51.518ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2532f642-d7f5-4b79-b0c8-1540df6f37c8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90822025-04-30T06:35:51.518ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
90832025-04-30T06:35:51.519ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee402481-d73a-4ac5-b3c0-f0662245676f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90842025-04-30T06:35:51.519ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
90852025-04-30T06:35:51.520ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2626bbc-fe65-4b1c-9c5c-9c7750d45e26", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90862025-04-30T06:35:51.520ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
90872025-04-30T06:35:51.521ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe6d7037-61a3-4e62-9c09-ae4702d47fa2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90882025-04-30T06:35:51.521ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
90892025-04-30T06:35:51.524ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "02d4a01a-54ef-4f34-ba78-a852eeb0cf2e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90902025-04-30T06:35:51.524ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
90912025-04-30T06:35:51.526ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
90922025-04-30T06:35:51.526ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
90932025-04-30T06:35:51.526ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
90942025-04-30T06:35:51.526ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
90952025-04-30T06:35:51.527ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
90962025-04-30T06:35:51.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dff330a-4fb9-4403-8565-8ad24d3dfe9b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90972025-04-30T06:35:51.527ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
90982025-04-30T06:35:51.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "37f2717d-ddec-4b83-aed3-53a8123ebe6a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
90992025-04-30T06:35:51.530ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
91002025-04-30T06:35:51.531ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91012025-04-30T06:35:51.536ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91022025-04-30T06:35:51.536ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91032025-04-30T06:35:51.536ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91042025-04-30T06:35:51.536ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91052025-04-30T06:35:51.539ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91062025-04-30T06:35:51.539ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91072025-04-30T06:35:51.539ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91082025-04-30T06:35:51.540ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf70390b-2859-4297-98a3-75e535e60e7b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91092025-04-30T06:35:51.540ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
91102025-04-30T06:35:51.542ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0e670e9-9d4a-4168-8c61-1714b5cd702b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91112025-04-30T06:35:51.542ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
91122025-04-30T06:35:51.544ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1039a302-1956-4282-9050-07674d6e9e40", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91132025-04-30T06:35:51.544ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
91142025-04-30T06:35:51.546ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5776f763-1129-4420-9642-ad0b0de3a3ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91152025-04-30T06:35:51.546ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
91162025-04-30T06:35:51.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dee2a0e-4fe7-4291-b70c-541ced46cc46", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91172025-04-30T06:35:51.556ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
91182025-04-30T06:35:51.558ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd5b3b2a-1265-4c89-9b6a-2f6229cc36b3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91192025-04-30T06:35:51.558ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
91202025-04-30T06:35:51.560ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9615482b-aae4-46ff-8ba0-cf0f2feef1ea", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91212025-04-30T06:35:51.560ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
91222025-04-30T06:35:51.562ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "96d055b2-540d-48d9-ab41-9020ba9856eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91232025-04-30T06:35:51.563ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
91242025-04-30T06:35:51.565ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed44fd7d-ce33-4395-aedb-421c647083a8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91252025-04-30T06:35:51.565ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
91262025-04-30T06:35:51.565ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91272025-04-30T06:35:51.565ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91282025-04-30T06:35:51.565ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91292025-04-30T06:35:51.566ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91302025-04-30T06:35:51.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91312025-04-30T06:35:51.567ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91322025-04-30T06:35:51.567ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
91332025-04-30T06:35:51.568ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
91342025-04-30T06:35:51.568ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91352025-04-30T06:35:51.572ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
91362025-04-30T06:35:51.572ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
91372025-04-30T06:35:51.572ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91382025-04-30T06:35:51.572ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91392025-04-30T06:35:51.572ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91402025-04-30T06:35:51.573ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91412025-04-30T06:35:51.576ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91422025-04-30T06:35:51.581ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
91432025-04-30T06:35:51.581ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91442025-04-30T06:35:51.581ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91452025-04-30T06:35:51.581ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91462025-04-30T06:35:51.582ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
91472025-04-30T06:35:51.582ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
91482025-04-30T06:35:51.633ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
91492025-04-30T06:35:51.633ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
91502025-04-30T06:35:51.635ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:34999 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
91512025-04-30T06:35:51.636ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
91522025-04-30T06:35:51.636ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
91532025-04-30T06:35:51.636ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:34999 req_id = c08c7cb4-75d2-4722-bbf4-94193993516f sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
91542025-04-30T06:35:51.637ZINFOtest_metrics (dropshot): request completed latency_us = 466 local_addr = [::1]:56237 method = GET remote_addr = [::1]:34999 req_id = c08c7cb4-75d2-4722-bbf4-94193993516f response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
91552025-04-30T06:35:51.637ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
91562025-04-30T06:35:51.638ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "c08c7cb4-75d2-4722-bbf4-94193993516f", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:50 GMT"} })
91572025-04-30T06:35:51.640ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
91582025-04-30T06:35:51.641ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
91592025-04-30T06:35:51.641ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
91602025-04-30T06:35:51.643ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
91612025-04-30T06:35:51.643ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91622025-04-30T06:35:51.643ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91632025-04-30T06:35:51.643ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91642025-04-30T06:35:51.644ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91652025-04-30T06:35:51.647ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91662025-04-30T06:35:51.652ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
91672025-04-30T06:35:51.652ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91682025-04-30T06:35:51.652ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91692025-04-30T06:35:51.652ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91702025-04-30T06:35:51.668ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
91712025-04-30T06:35:51.679ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
91722025-04-30T06:35:51.680ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
91732025-04-30T06:35:51.680ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 354.374057ms iteration = 30 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
91742025-04-30T06:35:51.680ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 31 reason = Signaled
91752025-04-30T06:35:51.680ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
91762025-04-30T06:35:51.680ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91772025-04-30T06:35:51.680ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91782025-04-30T06:35:51.680ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91792025-04-30T06:35:51.681ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91802025-04-30T06:35:51.683ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91812025-04-30T06:35:51.688ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
91822025-04-30T06:35:51.688ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
91832025-04-30T06:35:51.688ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
91842025-04-30T06:35:51.688ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
91852025-04-30T06:35:51.689ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
91862025-04-30T06:35:51.689ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
91872025-04-30T06:35:51.689ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91882025-04-30T06:35:51.689ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91892025-04-30T06:35:51.689ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91902025-04-30T06:35:51.692ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
91912025-04-30T06:35:51.692ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
91922025-04-30T06:35:51.693ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
91932025-04-30T06:35:51.700ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
91942025-04-30T06:35:51.700ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
91952025-04-30T06:35:51.701ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
91962025-04-30T06:35:51.701ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
91972025-04-30T06:35:51.702ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 30 4a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 0J..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49264 req_id = 787c1ad5-cbf9-407b-ad56-06b2f1934774
91982025-04-30T06:35:51.702ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 12362,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49264 req_id = 787c1ad5-cbf9-407b-ad56-06b2f1934774
91992025-04-30T06:35:51.702ZDEBGtest_metrics (store): query key key = _mgs._tcp
92002025-04-30T06:35:51.702ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92012025-04-30T06:35:51.702ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49264 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 787c1ad5-cbf9-407b-ad56-06b2f1934774
92022025-04-30T06:35:51.703ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3816, tv_nsec: 502497548 } })
92032025-04-30T06:35:51.703ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
92042025-04-30T06:35:51.703ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5a 9f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Z...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63409 req_id = 23b6a04f-123c-4c59-9801-659277a0d5d2
92052025-04-30T06:35:51.704ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23199,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63409 req_id = 23b6a04f-123c-4c59-9801-659277a0d5d2
92062025-04-30T06:35:51.704ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
92072025-04-30T06:35:51.704ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63409 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 23b6a04f-123c-4c59-9801-659277a0d5d2
92082025-04-30T06:35:51.749ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
92092025-04-30T06:35:51.750ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92102025-04-30T06:35:51.750ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
92112025-04-30T06:35:51.751ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92122025-04-30T06:35:51.751ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0e 2c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .,..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:63926 req_id = bee6cad3-3cb9-4431-8153-05f14dcec74c
92132025-04-30T06:35:51.751ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3628,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63926 req_id = bee6cad3-3cb9-4431-8153-05f14dcec74c
92142025-04-30T06:35:51.751ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
92152025-04-30T06:35:51.751ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3628, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:63926 req_id = bee6cad3-3cb9-4431-8153-05f14dcec74c
92162025-04-30T06:35:51.752ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
92172025-04-30T06:35:51.752ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
92182025-04-30T06:35:51.752ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
92192025-04-30T06:35:51.754ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "5aa589c0-61e5-4c4f-acde-de20b8cafb25", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92202025-04-30T06:35:51.754ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
92212025-04-30T06:35:51.756ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac95e521-4608-4c76-88f2-04e97b548db0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92222025-04-30T06:35:51.756ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
92232025-04-30T06:35:51.759ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1108879-5208-4994-895a-5fe35ab6b9b1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92242025-04-30T06:35:51.760ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
92252025-04-30T06:35:51.762ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7957fe1f-3282-43e9-aa67-7a89a92f7d55", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92262025-04-30T06:35:51.762ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
92272025-04-30T06:35:51.764ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "014858f0-2813-472b-a08d-5f30da15aa96", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92282025-04-30T06:35:51.764ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
92292025-04-30T06:35:51.765ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c3ed2bb-c8cc-47ee-a194-f7e59dd5a995", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92302025-04-30T06:35:51.766ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
92312025-04-30T06:35:51.768ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e1b80da6-a6b4-4bec-bc9a-0034bba226a1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92322025-04-30T06:35:51.768ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
92332025-04-30T06:35:51.770ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "59714710-5884-465c-9862-4d3116dd31ec", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92342025-04-30T06:35:51.770ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
92352025-04-30T06:35:51.772ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06694ad3-e1f6-442c-aa2d-815b5b4b995f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92362025-04-30T06:35:51.772ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
92372025-04-30T06:35:51.774ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7dd8ec3c-b233-4b56-8f21-71fa3a9d10e5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92382025-04-30T06:35:51.774ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
92392025-04-30T06:35:51.776ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b090cc47-72e1-4fef-8f03-9cb56cf32a25", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92402025-04-30T06:35:51.776ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
92412025-04-30T06:35:51.778ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "275e2a03-a13b-41a4-a771-a0c8a6c7a40d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92422025-04-30T06:35:51.778ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
92432025-04-30T06:35:51.781ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a831dbca-d048-4b07-9105-c4b77db66020", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92442025-04-30T06:35:51.782ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
92452025-04-30T06:35:51.782ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
92462025-04-30T06:35:51.784ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd579fb5-ca9c-4826-80ba-e1bb4ff91bc7", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92472025-04-30T06:35:51.785ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
92482025-04-30T06:35:51.787ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a13e701-d4e4-4bb0-9e00-a3c7dd1926e6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92492025-04-30T06:35:51.787ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
92502025-04-30T06:35:51.789ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "321be053-25da-4deb-977c-36d25b2b6425", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92512025-04-30T06:35:51.789ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
92522025-04-30T06:35:51.790ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "728a1786-440c-4a2c-af01-9e75c99443f3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92532025-04-30T06:35:51.790ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
92542025-04-30T06:35:51.792ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad39bf5c-1e80-4bc2-aac6-02933948c099", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92552025-04-30T06:35:51.792ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
92562025-04-30T06:35:51.794ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec4c4869-9e5d-4ed6-be0b-d1901f4ca291", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92572025-04-30T06:35:51.794ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
92582025-04-30T06:35:51.795ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b13781ef-682c-49c8-bb98-61facd00723a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92592025-04-30T06:35:51.795ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
92602025-04-30T06:35:51.796ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a781e03f-6edd-48b5-88ae-96a1413b70b7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92612025-04-30T06:35:51.796ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
92622025-04-30T06:35:51.797ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "470729d8-9c51-434c-b4d6-da644191b820", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92632025-04-30T06:35:51.797ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
92642025-04-30T06:35:51.798ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a7d13e4-d487-42b9-860d-e5fe944b7050", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92652025-04-30T06:35:51.798ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
92662025-04-30T06:35:51.799ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "81503123-d491-44cb-895b-cc35957cd46c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92672025-04-30T06:35:51.800ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
92682025-04-30T06:35:51.801ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "efa014a9-7d6c-428a-89fc-2bdb8a5883fd", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92692025-04-30T06:35:51.801ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
92702025-04-30T06:35:51.803ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "647e6914-d773-49f5-817f-865ab2b1a4d2", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92712025-04-30T06:35:51.803ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
92722025-04-30T06:35:51.805ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1210b7ab-70a3-4d50-8185-b111cdeb2edb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92732025-04-30T06:35:51.805ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
92742025-04-30T06:35:51.809ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aac1398a-6283-477e-8fb8-e03f3c4957da", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92752025-04-30T06:35:51.809ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
92762025-04-30T06:35:51.811ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf121872-0fba-4714-b27f-a637b852b4ef", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92772025-04-30T06:35:51.811ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
92782025-04-30T06:35:51.816ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e9f037-3af4-4a95-80ed-b9645d0907ea", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92792025-04-30T06:35:51.816ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
92802025-04-30T06:35:51.819ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29c3edd9-641e-499e-8a3f-dd43e810b554", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92812025-04-30T06:35:51.822ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
92822025-04-30T06:35:51.825ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0792c705-6812-40b3-a4c3-2e8ad810f457", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92832025-04-30T06:35:51.826ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
92842025-04-30T06:35:51.834ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "348354c8-1648-4f93-b6fa-f8120a766482", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92852025-04-30T06:35:51.834ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
92862025-04-30T06:35:51.835ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
92872025-04-30T06:35:51.835ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
92882025-04-30T06:35:51.835ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92892025-04-30T06:35:51.835ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
92902025-04-30T06:35:51.835ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92912025-04-30T06:35:51.846ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fd541d08-e0a7-4849-8759-c437b237b34f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
92922025-04-30T06:35:51.847ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
92932025-04-30T06:35:51.847ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
92942025-04-30T06:35:51.852ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
92952025-04-30T06:35:51.887ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92962025-04-30T06:35:51.887ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
92972025-04-30T06:35:51.890ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
92982025-04-30T06:35:51.900ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
92992025-04-30T06:35:51.900ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93002025-04-30T06:35:51.901ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93012025-04-30T06:35:51.901ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdac926e-1a7c-4cd9-b230-1a699073692c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93022025-04-30T06:35:51.902ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
93032025-04-30T06:35:51.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "059cdcfb-0159-4223-acf4-142ca2ab3d09", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93042025-04-30T06:35:51.904ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
93052025-04-30T06:35:51.907ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1a5164-c36d-48dc-ac80-de19c8e3967a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93062025-04-30T06:35:51.910ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
93072025-04-30T06:35:51.922ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb3ade50-aa14-4a63-afdc-767e8147c48b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93082025-04-30T06:35:51.923ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
93092025-04-30T06:35:51.933ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c71e730b-6eae-4655-9975-cb64d6f70eec", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93102025-04-30T06:35:51.934ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
93112025-04-30T06:35:51.939ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c04b97bd-c4b3-4595-a839-4615d985350c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93122025-04-30T06:35:51.940ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
93132025-04-30T06:35:51.946ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "798a8969-c106-4117-a9cd-a834f0e1e543", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93142025-04-30T06:35:51.947ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
93152025-04-30T06:35:51.948ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92860284-173e-450d-953a-689117e72e03", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93162025-04-30T06:35:51.951ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
93172025-04-30T06:35:51.956ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f505d23-84a2-4cc1-917c-b2564bfdba8e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93182025-04-30T06:35:51.957ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
93192025-04-30T06:35:51.961ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
93202025-04-30T06:35:51.962ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "057dae65-3162-46c9-9627-8528c41b1c73", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93212025-04-30T06:35:51.962ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
93222025-04-30T06:35:51.964ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "15f1259f-437e-4271-8dfa-b3c9b2473075", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93232025-04-30T06:35:51.964ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
93242025-04-30T06:35:51.964ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93252025-04-30T06:35:51.964ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93262025-04-30T06:35:51.964ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93272025-04-30T06:35:51.971ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93282025-04-30T06:35:51.972ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93292025-04-30T06:35:51.973ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
93302025-04-30T06:35:51.973ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
93312025-04-30T06:35:51.973ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93322025-04-30T06:35:51.973ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93332025-04-30T06:35:51.973ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93342025-04-30T06:35:51.973ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93352025-04-30T06:35:51.994ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93362025-04-30T06:35:51.997ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
93372025-04-30T06:35:51.997ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93382025-04-30T06:35:51.997ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93392025-04-30T06:35:51.998ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93402025-04-30T06:35:52.117ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
93412025-04-30T06:35:52.118ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
93422025-04-30T06:35:52.118ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
93432025-04-30T06:35:52.118ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93442025-04-30T06:35:52.118ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93452025-04-30T06:35:52.118ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93462025-04-30T06:35:52.119ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93472025-04-30T06:35:52.119ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:59950 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
93482025-04-30T06:35:52.120ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:59950 req_id = 0165ad7f-ac9a-4929-bc5a-432198a66011 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
93492025-04-30T06:35:52.121ZINFOtest_metrics (dropshot): request completed latency_us = 528 local_addr = [::1]:56237 method = GET remote_addr = [::1]:59950 req_id = 0165ad7f-ac9a-4929-bc5a-432198a66011 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
93502025-04-30T06:35:52.121ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0165ad7f-ac9a-4929-bc5a-432198a66011", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
93512025-04-30T06:35:52.123ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
93522025-04-30T06:35:52.123ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
93532025-04-30T06:35:52.123ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
93542025-04-30T06:35:52.126ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93552025-04-30T06:35:52.126ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93562025-04-30T06:35:52.126ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93572025-04-30T06:35:52.126ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93582025-04-30T06:35:52.127ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93592025-04-30T06:35:52.127ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93602025-04-30T06:35:52.135ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
93612025-04-30T06:35:52.135ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93622025-04-30T06:35:52.135ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93632025-04-30T06:35:52.136ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93642025-04-30T06:35:52.139ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93652025-04-30T06:35:52.143ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93662025-04-30T06:35:52.143ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93672025-04-30T06:35:52.143ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93682025-04-30T06:35:52.144ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93692025-04-30T06:35:52.145ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
93702025-04-30T06:35:52.145ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
93712025-04-30T06:35:52.146ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
93722025-04-30T06:35:52.192ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
93732025-04-30T06:35:52.192ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
93742025-04-30T06:35:52.192ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 512.560904ms iteration = 31 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
93752025-04-30T06:35:52.192ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 32 reason = Signaled
93762025-04-30T06:35:52.192ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
93772025-04-30T06:35:52.193ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
93782025-04-30T06:35:52.193ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93792025-04-30T06:35:52.193ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93802025-04-30T06:35:52.197ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93812025-04-30T06:35:52.201ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
93822025-04-30T06:35:52.206ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
93832025-04-30T06:35:52.206ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
93842025-04-30T06:35:52.206ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
93852025-04-30T06:35:52.207ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
93862025-04-30T06:35:52.212ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
93872025-04-30T06:35:52.212ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
93882025-04-30T06:35:52.213ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0d 5e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .^..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51812 req_id = 2a377b14-c417-4624-8a30-9786dd361757
93892025-04-30T06:35:52.213ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3422,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51812 req_id = 2a377b14-c417-4624-8a30-9786dd361757
93902025-04-30T06:35:52.213ZDEBGtest_metrics (store): query key key = _mgs._tcp
93912025-04-30T06:35:52.214ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93922025-04-30T06:35:52.214ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51812 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 2a377b14-c417-4624-8a30-9786dd361757
93932025-04-30T06:35:52.214ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
93942025-04-30T06:35:52.215ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3817, tv_nsec: 14053540 } })
93952025-04-30T06:35:52.215ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
93962025-04-30T06:35:52.216ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a9 67 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .g..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64892 req_id = 49afb352-3230-4802-8243-eecaa83da967
93972025-04-30T06:35:52.216ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43367,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64892 req_id = 49afb352-3230-4802-8243-eecaa83da967
93982025-04-30T06:35:52.216ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
93992025-04-30T06:35:52.216ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64892 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 49afb352-3230-4802-8243-eecaa83da967
94002025-04-30T06:35:52.267ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
94012025-04-30T06:35:52.268ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94022025-04-30T06:35:52.268ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94032025-04-30T06:35:52.268ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94042025-04-30T06:35:52.268ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94052025-04-30T06:35:52.269ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94062025-04-30T06:35:52.270ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 0e 81 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:54329 req_id = 8cb9ac8e-db80-4caf-866b-070654286aa3
94072025-04-30T06:35:52.270ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 3713,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54329 req_id = 8cb9ac8e-db80-4caf-866b-070654286aa3
94082025-04-30T06:35:52.270ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
94092025-04-30T06:35:52.270ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 3713, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:54329 req_id = 8cb9ac8e-db80-4caf-866b-070654286aa3
94102025-04-30T06:35:52.271ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
94112025-04-30T06:35:52.271ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
94122025-04-30T06:35:52.271ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
94132025-04-30T06:35:52.280ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94142025-04-30T06:35:52.297ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94152025-04-30T06:35:52.297ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94162025-04-30T06:35:52.297ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94172025-04-30T06:35:52.298ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94182025-04-30T06:35:52.301ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0dbbffa-bbb8-4614-b675-1d9bdb882bdc", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94192025-04-30T06:35:52.311ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
94202025-04-30T06:35:52.311ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94212025-04-30T06:35:52.311ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94222025-04-30T06:35:52.311ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94232025-04-30T06:35:52.311ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc872f21-4476-417c-a6b6-3c3d1fd69c64", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94242025-04-30T06:35:52.312ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
94252025-04-30T06:35:52.312ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "45388025-9eb5-489f-bd58-5b667b179f02", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94262025-04-30T06:35:52.312ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
94272025-04-30T06:35:52.315ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "236fc145-75d0-4335-bd08-0161530bd22b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94282025-04-30T06:35:52.315ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
94292025-04-30T06:35:52.321ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0415d03-8515-487d-87f9-ae500b1919c2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94302025-04-30T06:35:52.321ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
94312025-04-30T06:35:52.324ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f165e156-6567-4a8c-acba-518f2681b0af", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94322025-04-30T06:35:52.324ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
94332025-04-30T06:35:52.326ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e13db987-8d57-41e6-acb8-e30b5c8c13a6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94342025-04-30T06:35:52.326ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
94352025-04-30T06:35:52.348ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e62d31e-1b12-43b0-a2f6-dbb0f5e1548d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94362025-04-30T06:35:52.348ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
94372025-04-30T06:35:52.369ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3478d7ad-2984-4fb4-8758-de44275afd56", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94382025-04-30T06:35:52.369ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
94392025-04-30T06:35:52.370ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
94402025-04-30T06:35:52.379ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5454d9e9-daaf-4ab6-8280-988564b4af01", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94412025-04-30T06:35:52.379ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
94422025-04-30T06:35:52.381ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4237016b-deda-41e2-8ce5-b613ff5367eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94432025-04-30T06:35:52.382ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
94442025-04-30T06:35:52.383ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "886ba11d-b42a-4e83-a225-233667bc4745", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94452025-04-30T06:35:52.384ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
94462025-04-30T06:35:52.385ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1428ecd3-e388-4572-8369-614c90566265", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94472025-04-30T06:35:52.386ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
94482025-04-30T06:35:52.388ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "334fbe4d-5af8-4511-b9dc-c0195969ed23", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94492025-04-30T06:35:52.388ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
94502025-04-30T06:35:52.390ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "849393c2-e286-428f-9a10-f7181d59d815", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94512025-04-30T06:35:52.390ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
94522025-04-30T06:35:52.392ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4969264-3fe3-4bd0-9e19-4bceb1c2c281", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94532025-04-30T06:35:52.392ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
94542025-04-30T06:35:52.397ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d7046ee-4009-43b3-a926-f5e34ba81ac8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94552025-04-30T06:35:52.398ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
94562025-04-30T06:35:52.400ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0889b19f-d29a-49ed-93a9-c25316b0c069", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94572025-04-30T06:35:52.400ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
94582025-04-30T06:35:52.402ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61c2b9cc-2cba-4671-8c48-1fdbdd28fb58", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94592025-04-30T06:35:52.402ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
94602025-04-30T06:35:52.407ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07076d09-923c-4340-82cd-30c48c135fb1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94612025-04-30T06:35:52.407ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
94622025-04-30T06:35:52.409ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e2bd164a-688f-4055-8fd6-7b1d666e1a8a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94632025-04-30T06:35:52.409ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
94642025-04-30T06:35:52.410ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "60038e91-4ddc-49ac-89ff-df2d8a3c9647", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94652025-04-30T06:35:52.410ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
94662025-04-30T06:35:52.411ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8657165-2d48-473b-a062-a27734f403c1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94672025-04-30T06:35:52.411ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
94682025-04-30T06:35:52.413ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c319fe0-fa2b-43eb-87db-a9ce5ee83eb8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94692025-04-30T06:35:52.413ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
94702025-04-30T06:35:52.416ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8309de7-d3d3-4ea4-903a-0b55ad84c0ef", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94712025-04-30T06:35:52.416ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
94722025-04-30T06:35:52.419ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e1289c7-7efb-4214-8340-6e79bc5f2c9a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94732025-04-30T06:35:52.419ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
94742025-04-30T06:35:52.421ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
94752025-04-30T06:35:52.421ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
94762025-04-30T06:35:52.421ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94772025-04-30T06:35:52.421ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94782025-04-30T06:35:52.422ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94792025-04-30T06:35:52.423ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85348318-e590-4eb5-ba53-39ab2c7f647e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94802025-04-30T06:35:52.423ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
94812025-04-30T06:35:52.425ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
94822025-04-30T06:35:52.431ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
94832025-04-30T06:35:52.431ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94842025-04-30T06:35:52.431ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94852025-04-30T06:35:52.432ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94862025-04-30T06:35:52.434ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4c4c2d6-bc04-438e-b061-3340b2cedaab", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94872025-04-30T06:35:52.434ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
94882025-04-30T06:35:52.435ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
94892025-04-30T06:35:52.435ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
94902025-04-30T06:35:52.436ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
94912025-04-30T06:35:52.438ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6d78195-0b14-439a-b597-3f8715964890", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94922025-04-30T06:35:52.438ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
94932025-04-30T06:35:52.442ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3baaf318-1354-47ed-97f3-35d91ec19284", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94942025-04-30T06:35:52.442ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
94952025-04-30T06:35:52.442ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ceff975b-8a74-484f-bb94-1929e921de88", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94962025-04-30T06:35:52.443ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
94972025-04-30T06:35:52.444ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54ee6337-5c5f-4788-afa9-d82fae671690", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
94982025-04-30T06:35:52.445ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
94992025-04-30T06:35:52.446ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8adc263d-a5df-4f83-8168-f12e41ec9821", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95002025-04-30T06:35:52.446ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
95012025-04-30T06:35:52.447ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b73b4062-8d41-464e-8f39-91a1ff80fb7a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95022025-04-30T06:35:52.448ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
95032025-04-30T06:35:52.450ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e55378ac-b15f-4242-bb10-bedec2e9e44a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95042025-04-30T06:35:52.451ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
95052025-04-30T06:35:52.454ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a964698-b12a-48b6-9ac1-0d6da9693d2e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95062025-04-30T06:35:52.454ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
95072025-04-30T06:35:52.459ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e42173cc-8d0f-49de-ab63-0027d34b7340", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95082025-04-30T06:35:52.460ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
95092025-04-30T06:35:52.464ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2f77d66-1cd7-4d01-acbf-629886814640", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95102025-04-30T06:35:52.464ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
95112025-04-30T06:35:52.465ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
95122025-04-30T06:35:52.469ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e4e225f0-3c1b-444a-8d6b-87dc2ec98b56", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95132025-04-30T06:35:52.469ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
95142025-04-30T06:35:52.478ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc67cf3-9bb6-44dc-b0cd-7837ccc59b73", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95152025-04-30T06:35:52.479ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
95162025-04-30T06:35:52.485ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c7bb643-97cd-4425-bc11-7725da4f6df7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95172025-04-30T06:35:52.485ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
95182025-04-30T06:35:52.486ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "833f9868-fc64-4132-91bb-f88022ddda61", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95192025-04-30T06:35:52.486ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
95202025-04-30T06:35:52.489ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1484339-bc8b-45f5-a40a-cc938547e0a6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95212025-04-30T06:35:52.490ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
95222025-04-30T06:35:52.492ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5a70ac3d-c92f-4eee-bee1-32d5f88854d9", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95232025-04-30T06:35:52.492ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
95242025-04-30T06:35:52.494ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4d4e114-9bac-412f-891b-f8f8f2abb79c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95252025-04-30T06:35:52.494ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
95262025-04-30T06:35:52.494ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95272025-04-30T06:35:52.494ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95282025-04-30T06:35:52.494ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95292025-04-30T06:35:52.495ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95302025-04-30T06:35:52.498ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95312025-04-30T06:35:52.503ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95322025-04-30T06:35:52.503ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
95332025-04-30T06:35:52.503ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95342025-04-30T06:35:52.503ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95352025-04-30T06:35:52.503ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95362025-04-30T06:35:52.504ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95372025-04-30T06:35:52.507ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95382025-04-30T06:35:52.512ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
95392025-04-30T06:35:52.512ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95402025-04-30T06:35:52.512ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95412025-04-30T06:35:52.512ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95422025-04-30T06:35:52.513ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95432025-04-30T06:35:52.514ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95442025-04-30T06:35:52.517ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
95452025-04-30T06:35:52.517ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95462025-04-30T06:35:52.517ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95472025-04-30T06:35:52.517ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95482025-04-30T06:35:52.518ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95492025-04-30T06:35:52.519ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95502025-04-30T06:35:52.519ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95512025-04-30T06:35:52.520ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95522025-04-30T06:35:52.520ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
95532025-04-30T06:35:52.573ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
95542025-04-30T06:35:52.574ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
95552025-04-30T06:35:52.574ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:55146 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
95562025-04-30T06:35:52.575ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95572025-04-30T06:35:52.575ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95582025-04-30T06:35:52.575ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
95592025-04-30T06:35:52.576ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
95602025-04-30T06:35:52.578ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95612025-04-30T06:35:52.582ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
95622025-04-30T06:35:52.582ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95632025-04-30T06:35:52.582ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95642025-04-30T06:35:52.582ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95652025-04-30T06:35:52.582ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:55146 req_id = 6c6c29fc-f540-4e0a-be0b-c33d309e072c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95662025-04-30T06:35:52.583ZINFOtest_metrics (dropshot): request completed latency_us = 632 local_addr = [::1]:56237 method = GET remote_addr = [::1]:55146 req_id = 6c6c29fc-f540-4e0a-be0b-c33d309e072c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
95672025-04-30T06:35:52.584ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c6c29fc-f540-4e0a-be0b-c33d309e072c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:51 GMT"} })
95682025-04-30T06:35:52.586ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95692025-04-30T06:35:52.586ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
95702025-04-30T06:35:52.586ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
95712025-04-30T06:35:52.588ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95722025-04-30T06:35:52.588ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95732025-04-30T06:35:52.588ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95742025-04-30T06:35:52.588ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95752025-04-30T06:35:52.588ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95762025-04-30T06:35:52.589ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
95772025-04-30T06:35:52.589ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
95782025-04-30T06:35:52.590ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
95792025-04-30T06:35:52.590ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
95802025-04-30T06:35:52.592ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
95812025-04-30T06:35:52.592ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95822025-04-30T06:35:52.597ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
95832025-04-30T06:35:52.597ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95842025-04-30T06:35:52.597ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95852025-04-30T06:35:52.597ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95862025-04-30T06:35:52.599ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
95872025-04-30T06:35:52.599ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
95882025-04-30T06:35:52.629ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
95892025-04-30T06:35:52.629ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
95902025-04-30T06:35:52.629ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 436.589056ms iteration = 32 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
95912025-04-30T06:35:52.629ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 33 reason = Signaled
95922025-04-30T06:35:52.629ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
95932025-04-30T06:35:52.629ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
95942025-04-30T06:35:52.629ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
95952025-04-30T06:35:52.629ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
95962025-04-30T06:35:52.631ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
95972025-04-30T06:35:52.634ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
95982025-04-30T06:35:52.635ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
95992025-04-30T06:35:52.639ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
96002025-04-30T06:35:52.639ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
96012025-04-30T06:35:52.639ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
96022025-04-30T06:35:52.641ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
96032025-04-30T06:35:52.644ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
96042025-04-30T06:35:52.644ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
96052025-04-30T06:35:52.645ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: bf 26 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .&..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:59672 req_id = 6f177ef6-b171-4bcf-850b-de883ea9e223
96062025-04-30T06:35:52.645ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48934,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59672 req_id = 6f177ef6-b171-4bcf-850b-de883ea9e223
96072025-04-30T06:35:52.646ZDEBGtest_metrics (store): query key key = _mgs._tcp
96082025-04-30T06:35:52.646ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96092025-04-30T06:35:52.646ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59672 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6f177ef6-b171-4bcf-850b-de883ea9e223
96102025-04-30T06:35:52.647ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3817, tv_nsec: 445953973 } })
96112025-04-30T06:35:52.647ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
96122025-04-30T06:35:52.647ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ed 93 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49879 req_id = 6f475183-5280-4c36-aa8e-6db5f8f054bf
96132025-04-30T06:35:52.647ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60819,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49879 req_id = 6f475183-5280-4c36-aa8e-6db5f8f054bf
96142025-04-30T06:35:52.647ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
96152025-04-30T06:35:52.647ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49879 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6f475183-5280-4c36-aa8e-6db5f8f054bf
96162025-04-30T06:35:52.736ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
96172025-04-30T06:35:52.736ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96182025-04-30T06:35:52.736ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96192025-04-30T06:35:52.736ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96202025-04-30T06:35:52.736ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96212025-04-30T06:35:52.737ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96222025-04-30T06:35:52.738ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
96232025-04-30T06:35:52.738ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: dc cf 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58045 req_id = a51d320c-d35b-497d-a457-74f45579b95b
96242025-04-30T06:35:52.739ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 56527,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58045 req_id = a51d320c-d35b-497d-a457-74f45579b95b
96252025-04-30T06:35:52.739ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
96262025-04-30T06:35:52.739ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 56527, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58045 req_id = a51d320c-d35b-497d-a457-74f45579b95b
96272025-04-30T06:35:52.739ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
96282025-04-30T06:35:52.739ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
96292025-04-30T06:35:52.739ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
96302025-04-30T06:35:52.741ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
96312025-04-30T06:35:52.748ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
96322025-04-30T06:35:52.748ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96332025-04-30T06:35:52.748ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96342025-04-30T06:35:52.749ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96352025-04-30T06:35:52.751ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "aad0429d-fc56-4b4e-bf53-0eeb769fe7db", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96362025-04-30T06:35:52.751ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
96372025-04-30T06:35:52.752ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96382025-04-30T06:35:52.752ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96392025-04-30T06:35:52.752ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96402025-04-30T06:35:52.756ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "504dd293-2b09-4fc2-96ad-afe63adbc5d5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96412025-04-30T06:35:52.757ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
96422025-04-30T06:35:52.773ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "46598b54-4418-4090-af99-805527a2fb91", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96432025-04-30T06:35:52.774ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
96442025-04-30T06:35:52.782ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4cfa64a-96e0-40fe-ac70-8ca284a2f18a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96452025-04-30T06:35:52.782ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
96462025-04-30T06:35:52.787ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a96d09c-2bc0-474b-872b-c54ff6ac8849", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96472025-04-30T06:35:52.787ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
96482025-04-30T06:35:52.796ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "15312397-a70d-4e3b-a721-d1c3196161a8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96492025-04-30T06:35:52.798ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
96502025-04-30T06:35:52.804ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
96512025-04-30T06:35:52.804ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9cf0a1bc-2402-4bda-81de-f67a9c64d8dc", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96522025-04-30T06:35:52.804ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
96532025-04-30T06:35:52.812ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f5e56aa-e84a-4e07-953c-19b63e1d6c07", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96542025-04-30T06:35:52.812ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
96552025-04-30T06:35:52.813ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0638a054-4b4a-4cff-b5f8-0fa2183dd858", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96562025-04-30T06:35:52.813ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
96572025-04-30T06:35:52.814ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "aae6b337-25d0-4c17-911b-6d8861f87543", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96582025-04-30T06:35:52.814ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
96592025-04-30T06:35:52.815ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9116dccc-6e2b-4da9-9292-f91f84fc73ad", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96602025-04-30T06:35:52.815ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
96612025-04-30T06:35:52.816ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f7d7a063-b2a9-4068-bf9c-f1c8d1ad365e", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96622025-04-30T06:35:52.816ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
96632025-04-30T06:35:52.817ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef32cff7-fdee-449d-ab6d-20d8d07657e3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96642025-04-30T06:35:52.818ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
96652025-04-30T06:35:52.820ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6a52f95-483a-41fa-84c7-3eeb5598103b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96662025-04-30T06:35:52.820ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
96672025-04-30T06:35:52.822ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6da8f08-c1cf-4f0d-811a-c5bd8c251deb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96682025-04-30T06:35:52.822ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
96692025-04-30T06:35:52.824ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aabec80-9493-459f-a302-0d03370ebd73", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96702025-04-30T06:35:52.824ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
96712025-04-30T06:35:52.827ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "23bef076-950a-469a-b4da-d4115f19f4e2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96722025-04-30T06:35:52.827ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
96732025-04-30T06:35:52.830ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2248dc8-444b-4960-900c-97f9c0d4427c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96742025-04-30T06:35:52.830ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
96752025-04-30T06:35:52.832ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e213d8d3-1ca3-4e08-94b4-08f9654dc55a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96762025-04-30T06:35:52.833ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
96772025-04-30T06:35:52.833ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14318a50-99db-43e3-86d4-4a079ec71078", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96782025-04-30T06:35:52.834ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
96792025-04-30T06:35:52.835ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77b6706a-7f0d-41bb-861b-9fd9e625e3bd", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96802025-04-30T06:35:52.835ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
96812025-04-30T06:35:52.837ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b2a8b09c-987a-48aa-ab32-cd6091cdfbef", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96822025-04-30T06:35:52.837ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
96832025-04-30T06:35:52.840ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e944f95b-bee1-4bfc-83ae-bad4a03c8e00", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96842025-04-30T06:35:52.840ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
96852025-04-30T06:35:52.842ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c43b66e-6d3b-45a0-a6c6-a2e345f654f5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96862025-04-30T06:35:52.842ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
96872025-04-30T06:35:52.845ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6efca098-08ab-4562-9aa3-5e096c273a78", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96882025-04-30T06:35:52.845ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
96892025-04-30T06:35:52.848ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6b33847-6a16-4f9e-bed8-48a4b377815f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96902025-04-30T06:35:52.848ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
96912025-04-30T06:35:52.851ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "02aff00b-c0ec-4c88-b2a1-6569f9c6430a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96922025-04-30T06:35:52.851ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
96932025-04-30T06:35:52.852ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
96942025-04-30T06:35:52.852ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
96952025-04-30T06:35:52.852ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
96962025-04-30T06:35:52.852ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
96972025-04-30T06:35:52.853ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
96982025-04-30T06:35:52.855ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e151e61-64fc-4f0b-90b8-62221eadef55", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
96992025-04-30T06:35:52.855ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
97002025-04-30T06:35:52.856ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97012025-04-30T06:35:52.862ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97022025-04-30T06:35:52.862ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97032025-04-30T06:35:52.862ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97042025-04-30T06:35:52.862ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97052025-04-30T06:35:52.866ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97062025-04-30T06:35:52.866ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97072025-04-30T06:35:52.866ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97082025-04-30T06:35:52.866ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d79ca53-2e45-4778-b589-26c63de37690", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97092025-04-30T06:35:52.866ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
97102025-04-30T06:35:52.870ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3889c107-f6b6-4c36-9345-7f7a09b3daba", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97112025-04-30T06:35:52.870ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
97122025-04-30T06:35:52.871ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9108abec-ae9f-4e59-ad1f-42e76092e37a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97132025-04-30T06:35:52.871ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
97142025-04-30T06:35:52.873ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eefb74f6-4e5f-48b2-b536-32b9148a11f9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97152025-04-30T06:35:52.873ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
97162025-04-30T06:35:52.874ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "06add35c-adce-4d43-8a16-97ed6729bc45", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97172025-04-30T06:35:52.874ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
97182025-04-30T06:35:52.876ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fa17e07-5724-43db-af1b-8ca39b3624e2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97192025-04-30T06:35:52.876ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
97202025-04-30T06:35:52.879ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f5fe6a-46e9-46ca-b8f5-2e8561378e01", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97212025-04-30T06:35:52.880ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
97222025-04-30T06:35:52.883ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "899cfb96-7378-4df3-ba86-86336703f134", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97232025-04-30T06:35:52.883ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
97242025-04-30T06:35:52.886ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "43e9e7e3-7e7c-4b77-942c-3ee31104a83d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97252025-04-30T06:35:52.886ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
97262025-04-30T06:35:52.889ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e46c7bd7-b54f-4c49-844c-0a229ad2eb7b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97272025-04-30T06:35:52.889ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
97282025-04-30T06:35:52.891ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d78b9f97-54ce-4052-b655-787583833c24", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97292025-04-30T06:35:52.891ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
97302025-04-30T06:35:52.894ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e49b40e8-cfb3-4da0-8a37-67c363566f4e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97312025-04-30T06:35:52.894ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
97322025-04-30T06:35:52.899ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f471e88-06d8-495b-83fd-80fc486bae50", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97332025-04-30T06:35:52.899ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
97342025-04-30T06:35:52.900ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
97352025-04-30T06:35:52.900ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "17a10c7e-038c-40af-8c9c-6fad4f2eca76", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97362025-04-30T06:35:52.900ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
97372025-04-30T06:35:52.902ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da47a3a4-a2a0-49e4-ac48-e25b8e000713", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97382025-04-30T06:35:52.902ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
97392025-04-30T06:35:52.903ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d3aaff2-d22f-4888-b8f8-58c4568e035c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97402025-04-30T06:35:52.903ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
97412025-04-30T06:35:52.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79e1b17e-9e66-4a60-907b-9ea129312c38", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97422025-04-30T06:35:52.904ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97432025-04-30T06:35:52.904ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97442025-04-30T06:35:52.904ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97452025-04-30T06:35:52.904ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97462025-04-30T06:35:52.905ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97472025-04-30T06:35:52.910ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97482025-04-30T06:35:52.914ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97492025-04-30T06:35:52.914ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
97502025-04-30T06:35:52.914ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97512025-04-30T06:35:52.914ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97522025-04-30T06:35:52.914ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97532025-04-30T06:35:52.915ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97542025-04-30T06:35:52.918ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97552025-04-30T06:35:52.922ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
97562025-04-30T06:35:52.923ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97572025-04-30T06:35:52.923ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97582025-04-30T06:35:52.924ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97592025-04-30T06:35:53.003ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
97602025-04-30T06:35:53.003ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
97612025-04-30T06:35:53.004ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
97622025-04-30T06:35:53.004ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97632025-04-30T06:35:53.004ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97642025-04-30T06:35:53.004ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97652025-04-30T06:35:53.005ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97662025-04-30T06:35:53.005ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:63213 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
97672025-04-30T06:35:53.006ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:63213 req_id = a5a5dc05-2b6d-4b09-a877-624d853178f5 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97682025-04-30T06:35:53.007ZINFOtest_metrics (dropshot): request completed latency_us = 1119 local_addr = [::1]:56237 method = GET remote_addr = [::1]:63213 req_id = a5a5dc05-2b6d-4b09-a877-624d853178f5 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
97692025-04-30T06:35:53.008ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5a5dc05-2b6d-4b09-a877-624d853178f5", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
97702025-04-30T06:35:53.016ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97712025-04-30T06:35:53.016ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
97722025-04-30T06:35:53.016ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
97732025-04-30T06:35:53.016ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97742025-04-30T06:35:53.017ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97752025-04-30T06:35:53.017ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97762025-04-30T06:35:53.017ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97772025-04-30T06:35:53.017ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97782025-04-30T06:35:53.017ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97792025-04-30T06:35:53.019ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
97802025-04-30T06:35:53.019ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97812025-04-30T06:35:53.019ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97822025-04-30T06:35:53.019ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97832025-04-30T06:35:53.021ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
97842025-04-30T06:35:53.028ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
97852025-04-30T06:35:53.028ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97862025-04-30T06:35:53.028ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97872025-04-30T06:35:53.029ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
97882025-04-30T06:35:53.029ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
97892025-04-30T06:35:53.030ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
97902025-04-30T06:35:53.030ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
97912025-04-30T06:35:53.065ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
97922025-04-30T06:35:53.065ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
97932025-04-30T06:35:53.065ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 436.156853ms iteration = 33 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
97942025-04-30T06:35:53.065ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 34 reason = Signaled
97952025-04-30T06:35:53.065ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
97962025-04-30T06:35:53.065ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
97972025-04-30T06:35:53.065ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
97982025-04-30T06:35:53.065ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
97992025-04-30T06:35:53.067ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98002025-04-30T06:35:53.068ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
98012025-04-30T06:35:53.071ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98022025-04-30T06:35:53.075ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
98032025-04-30T06:35:53.075ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
98042025-04-30T06:35:53.075ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
98052025-04-30T06:35:53.075ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
98062025-04-30T06:35:53.088ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
98072025-04-30T06:35:53.088ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
98082025-04-30T06:35:53.090ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 4c c3 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 L............_mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60330 req_id = b702e058-dbce-47c9-9978-5b6d638e0924
98092025-04-30T06:35:53.090ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19651,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60330 req_id = b702e058-dbce-47c9-9978-5b6d638e0924
98102025-04-30T06:35:53.090ZDEBGtest_metrics (store): query key key = _mgs._tcp
98112025-04-30T06:35:53.090ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98122025-04-30T06:35:53.090ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60330 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b702e058-dbce-47c9-9978-5b6d638e0924
98132025-04-30T06:35:53.091ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3817, tv_nsec: 890171748 } })
98142025-04-30T06:35:53.091ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
98152025-04-30T06:35:53.091ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: ce 73 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .s..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:54293 req_id = 7fca5dd9-3f5c-4773-ad26-4f90c59a30a2
98162025-04-30T06:35:53.091ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 52851,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54293 req_id = 7fca5dd9-3f5c-4773-ad26-4f90c59a30a2
98172025-04-30T06:35:53.091ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
98182025-04-30T06:35:53.091ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:54293 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7fca5dd9-3f5c-4773-ad26-4f90c59a30a2
98192025-04-30T06:35:53.179ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
98202025-04-30T06:35:53.180ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
98212025-04-30T06:35:53.180ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
98222025-04-30T06:35:53.180ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98232025-04-30T06:35:53.180ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98242025-04-30T06:35:53.182ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98252025-04-30T06:35:53.182ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b2 dd 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:62232 req_id = 3d612283-89cb-4242-bb15-cbd9e53d8174
98262025-04-30T06:35:53.182ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45789,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62232 req_id = 3d612283-89cb-4242-bb15-cbd9e53d8174
98272025-04-30T06:35:53.183ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
98282025-04-30T06:35:53.183ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45789, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:62232 req_id = 3d612283-89cb-4242-bb15-cbd9e53d8174
98292025-04-30T06:35:53.183ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
98302025-04-30T06:35:53.183ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
98312025-04-30T06:35:53.184ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
98322025-04-30T06:35:53.185ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
98332025-04-30T06:35:53.192ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
98342025-04-30T06:35:53.192ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98352025-04-30T06:35:53.192ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98362025-04-30T06:35:53.193ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98372025-04-30T06:35:53.197ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
98382025-04-30T06:35:53.197ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
98392025-04-30T06:35:53.198ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
98402025-04-30T06:35:53.202ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bbf1d62-c882-4c16-9ba8-7f9214b492a1", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98412025-04-30T06:35:53.204ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
98422025-04-30T06:35:53.206ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed61dfd6-8147-4867-bff3-3e1bc45051a0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98432025-04-30T06:35:53.206ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
98442025-04-30T06:35:53.209ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "69e457c3-7d2e-4e31-8417-f183b6a7e998", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98452025-04-30T06:35:53.209ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
98462025-04-30T06:35:53.211ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "901c8c85-7d90-480f-8d63-8b56d452555d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98472025-04-30T06:35:53.211ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
98482025-04-30T06:35:53.213ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1957d844-20d9-484d-a85b-64ce30a4ede6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98492025-04-30T06:35:53.213ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
98502025-04-30T06:35:53.217ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "438c206b-1832-4bbd-ad90-095919c62892", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98512025-04-30T06:35:53.217ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
98522025-04-30T06:35:53.220ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "28de968b-59d6-4922-b413-9f5fd883db0c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98532025-04-30T06:35:53.220ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
98542025-04-30T06:35:53.223ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a3fd8a1-4d77-4886-a27c-a944dd65e234", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98552025-04-30T06:35:53.223ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
98562025-04-30T06:35:53.226ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8dbb4a5f-abcf-40b8-b8a8-c88c0cd22830", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98572025-04-30T06:35:53.226ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
98582025-04-30T06:35:53.230ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0214f14-1fe0-4fc4-aaad-87c4c589a08e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98592025-04-30T06:35:53.230ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
98602025-04-30T06:35:53.232ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea7c35ba-4720-44ca-9f13-3e22a81d5cf4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98612025-04-30T06:35:53.233ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
98622025-04-30T06:35:53.234ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0498a831-dfad-4bfe-8db5-4ec1010b8e40", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98632025-04-30T06:35:53.234ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
98642025-04-30T06:35:53.236ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "930e1b83-3765-4bd9-aa04-7cb282e6b7e4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98652025-04-30T06:35:53.236ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
98662025-04-30T06:35:53.238ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82afa601-1102-4dee-9c62-778168ce2c97", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98672025-04-30T06:35:53.239ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
98682025-04-30T06:35:53.244ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
98692025-04-30T06:35:53.249ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3eb549c9-2fd9-4ee4-ae0c-f5fdd36538f6", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98702025-04-30T06:35:53.249ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
98712025-04-30T06:35:53.251ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d1a7f23-087d-4e90-9f04-e3e36b5b6cee", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98722025-04-30T06:35:53.251ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
98732025-04-30T06:35:53.253ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a35565e-d53d-4ea7-aaa2-42b61e512599", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98742025-04-30T06:35:53.253ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
98752025-04-30T06:35:53.259ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03a02e66-3ff7-4065-9abe-278bcdd4e892", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98762025-04-30T06:35:53.260ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
98772025-04-30T06:35:53.263ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d463b44f-3836-4caf-99ba-6a224b887ed4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98782025-04-30T06:35:53.264ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
98792025-04-30T06:35:53.266ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8b14161-03ae-4c66-9f6a-eace7877be6c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98802025-04-30T06:35:53.266ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
98812025-04-30T06:35:53.267ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "67adac77-ae04-4a08-babb-6f8503da1357", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98822025-04-30T06:35:53.267ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
98832025-04-30T06:35:53.268ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bac315a6-aa82-4b01-be5f-3c4944f84b37", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98842025-04-30T06:35:53.268ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
98852025-04-30T06:35:53.269ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3723a9d3-92f9-4663-bf49-2271ee97b2f8", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98862025-04-30T06:35:53.270ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
98872025-04-30T06:35:53.271ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c99d4f91-480e-45d2-b948-54a5d5f0258c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98882025-04-30T06:35:53.271ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
98892025-04-30T06:35:53.276ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef9c6757-76ef-4567-b93a-643fb4394c61", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98902025-04-30T06:35:53.276ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
98912025-04-30T06:35:53.280ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a92c2f6-3a39-49f2-aba9-64d6763fc63e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98922025-04-30T06:35:53.280ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
98932025-04-30T06:35:53.283ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "25a522c6-69cb-4c08-bbfa-1f6db4dfaf18", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98942025-04-30T06:35:53.283ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
98952025-04-30T06:35:53.285ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b51e4d58-11b6-46a9-b5eb-e11cb56a9620", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98962025-04-30T06:35:53.285ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
98972025-04-30T06:35:53.287ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d440494-81d7-470d-ad64-03e91ab18d67", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
98982025-04-30T06:35:53.288ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
98992025-04-30T06:35:53.290ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9887774-37a5-4ae8-942b-9fcaccfc0e7f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99002025-04-30T06:35:53.290ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
99012025-04-30T06:35:53.291ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fe46351-e90d-4d14-99df-5bdd1dd45c50", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99022025-04-30T06:35:53.291ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
99032025-04-30T06:35:53.292ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f42caae-74b7-4988-b550-e1ccebe100d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99042025-04-30T06:35:53.292ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
99052025-04-30T06:35:53.293ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4534e39a-6b15-4ee0-9b55-71dc775b3878", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99062025-04-30T06:35:53.294ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
99072025-04-30T06:35:53.299ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99082025-04-30T06:35:53.299ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99092025-04-30T06:35:53.299ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99102025-04-30T06:35:53.299ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99112025-04-30T06:35:53.300ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99122025-04-30T06:35:53.300ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
99132025-04-30T06:35:53.301ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99142025-04-30T06:35:53.301ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 33173a67-d5a5-44df-a2ea-7505ad6d3d9c uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99152025-04-30T06:35:53.302ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 33173a67-d5a5-44df-a2ea-7505ad6d3d9c resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99162025-04-30T06:35:53.302ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 33173a67-d5a5-44df-a2ea-7505ad6d3d9c roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99172025-04-30T06:35:53.302ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 33173a67-d5a5-44df-a2ea-7505ad6d3d9c resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99182025-04-30T06:35:53.303ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d00439e-dd99-4458-bb60-2f7439702ddf", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99192025-04-30T06:35:53.304ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
99202025-04-30T06:35:53.307ZINFOtest_metrics (dropshot_internal): request completed latency_us = 5472 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 33173a67-d5a5-44df-a2ea-7505ad6d3d9c response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
99212025-04-30T06:35:53.307ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99222025-04-30T06:35:53.313ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99232025-04-30T06:35:53.313ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99242025-04-30T06:35:53.313ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99252025-04-30T06:35:53.313ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99262025-04-30T06:35:53.314ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "33173a67-d5a5-44df-a2ea-7505ad6d3d9c", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99272025-04-30T06:35:53.315ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99282025-04-30T06:35:53.316ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83a62289-bf6e-4442-9d66-20dc226bfac9 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99292025-04-30T06:35:53.316ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83a62289-bf6e-4442-9d66-20dc226bfac9 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99302025-04-30T06:35:53.316ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83a62289-bf6e-4442-9d66-20dc226bfac9 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99312025-04-30T06:35:53.317ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83a62289-bf6e-4442-9d66-20dc226bfac9 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99322025-04-30T06:35:53.318ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e603fcc5-3e02-4223-9068-676b0198af0c", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99332025-04-30T06:35:53.319ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
99342025-04-30T06:35:53.319ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99352025-04-30T06:35:53.319ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99362025-04-30T06:35:53.320ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99372025-04-30T06:35:53.320ZINFOtest_metrics (dropshot_internal): request completed latency_us = 4179 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 83a62289-bf6e-4442-9d66-20dc226bfac9 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
99382025-04-30T06:35:53.321ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "83a62289-bf6e-4442-9d66-20dc226bfac9", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99392025-04-30T06:35:53.321ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
99402025-04-30T06:35:53.321ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99412025-04-30T06:35:53.321ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
99422025-04-30T06:35:53.325ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85f69014-ff26-47f2-9b11-d5e5f4e51e06", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99432025-04-30T06:35:53.325ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
99442025-04-30T06:35:53.328ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e50b7b2b-2e7b-4b3a-b1f1-fc8d6d260fea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99452025-04-30T06:35:53.328ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
99462025-04-30T06:35:53.371ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
99472025-04-30T06:35:53.383ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "85974bad-c835-4b19-9bab-b4c82dbb2939", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99482025-04-30T06:35:53.383ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
99492025-04-30T06:35:53.395ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5c9b593e-d7ff-4769-bc78-ead80c8abeee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99502025-04-30T06:35:53.395ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
99512025-04-30T06:35:53.410ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7f367233-d7e5-4b73-84a3-ad1bc9f7f5bb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99522025-04-30T06:35:53.411ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
99532025-04-30T06:35:53.414ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1db5d664-c2ab-4152-8083-05b680a7154b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99542025-04-30T06:35:53.415ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
99552025-04-30T06:35:53.416ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d208648b-0266-4c7b-81ae-dbc914b26933", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99562025-04-30T06:35:53.417ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
99572025-04-30T06:35:53.418ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "db003766-d991-415a-ab14-b3ea7d954c07", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99582025-04-30T06:35:53.418ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
99592025-04-30T06:35:53.419ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c87f7009-47d4-47b2-a077-0e30a192cd4c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99602025-04-30T06:35:53.419ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
99612025-04-30T06:35:53.420ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9f84e6d-f30f-420a-b243-1847fed427d1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
99622025-04-30T06:35:53.420ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99632025-04-30T06:35:53.420ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99642025-04-30T06:35:53.420ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99652025-04-30T06:35:53.420ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99662025-04-30T06:35:53.421ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99672025-04-30T06:35:53.422ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
99682025-04-30T06:35:53.422ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99692025-04-30T06:35:53.422ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99702025-04-30T06:35:53.422ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99712025-04-30T06:35:53.422ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99722025-04-30T06:35:53.425ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99732025-04-30T06:35:53.431ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
99742025-04-30T06:35:53.431ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99752025-04-30T06:35:53.431ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99762025-04-30T06:35:53.432ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99772025-04-30T06:35:53.432ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99782025-04-30T06:35:53.435ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99792025-04-30T06:35:53.435ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
99802025-04-30T06:35:53.435ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
99812025-04-30T06:35:53.435ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99822025-04-30T06:35:53.435ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99832025-04-30T06:35:53.437ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99842025-04-30T06:35:53.439ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
99852025-04-30T06:35:53.440ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
99862025-04-30T06:35:53.440ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
99872025-04-30T06:35:53.441ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
99882025-04-30T06:35:53.449ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
99892025-04-30T06:35:53.449ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
99902025-04-30T06:35:53.449ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
99912025-04-30T06:35:53.450ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
99922025-04-30T06:35:53.542ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
99932025-04-30T06:35:53.542ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
99942025-04-30T06:35:53.544ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:60871 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
99952025-04-30T06:35:53.544ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99962025-04-30T06:35:53.544ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
99972025-04-30T06:35:53.545ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:60871 req_id = 87717773-8911-4d17-b4e5-476fdb4fcd59 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99982025-04-30T06:35:53.547ZINFOtest_metrics (dropshot): request completed latency_us = 1908 local_addr = [::1]:56237 method = GET remote_addr = [::1]:60871 req_id = 87717773-8911-4d17-b4e5-476fdb4fcd59 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
99992025-04-30T06:35:53.547ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100002025-04-30T06:35:53.559ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100012025-04-30T06:35:53.559ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100022025-04-30T06:35:53.559ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
100032025-04-30T06:35:53.559ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "87717773-8911-4d17-b4e5-476fdb4fcd59", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:52 GMT"} })
100042025-04-30T06:35:53.562ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100052025-04-30T06:35:53.572ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
100062025-04-30T06:35:53.573ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
100072025-04-30T06:35:53.581ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100082025-04-30T06:35:53.581ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100092025-04-30T06:35:53.581ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100102025-04-30T06:35:53.581ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100112025-04-30T06:35:53.583ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100122025-04-30T06:35:53.593ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100132025-04-30T06:35:53.595ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100142025-04-30T06:35:53.622ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
100152025-04-30T06:35:53.630ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
100162025-04-30T06:35:53.639ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
100172025-04-30T06:35:53.639ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
100182025-04-30T06:35:53.640ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100192025-04-30T06:35:53.653ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
100202025-04-30T06:35:53.653ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100212025-04-30T06:35:53.654ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100222025-04-30T06:35:53.654ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100232025-04-30T06:35:53.660ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
100242025-04-30T06:35:53.660ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
100252025-04-30T06:35:53.711ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
100262025-04-30T06:35:53.723ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
100272025-04-30T06:35:53.733ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
100282025-04-30T06:35:53.733ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
100292025-04-30T06:35:53.733ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 667.932903ms iteration = 34 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
100302025-04-30T06:35:53.733ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 35 reason = Signaled
100312025-04-30T06:35:53.733ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
100322025-04-30T06:35:53.733ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100332025-04-30T06:35:53.733ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100342025-04-30T06:35:53.733ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100352025-04-30T06:35:53.734ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100362025-04-30T06:35:53.737ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100372025-04-30T06:35:53.744ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
100382025-04-30T06:35:53.744ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
100392025-04-30T06:35:53.744ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
100402025-04-30T06:35:53.744ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
100412025-04-30T06:35:53.749ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
100422025-04-30T06:35:53.749ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
100432025-04-30T06:35:53.750ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 00 c7 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:56008 req_id = 79286675-9b18-4d3b-a50d-a753a901a763
100442025-04-30T06:35:53.750ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 199,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56008 req_id = 79286675-9b18-4d3b-a50d-a753a901a763
100452025-04-30T06:35:53.750ZDEBGtest_metrics (store): query key key = _mgs._tcp
100462025-04-30T06:35:53.750ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
100472025-04-30T06:35:53.750ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:56008 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 79286675-9b18-4d3b-a50d-a753a901a763
100482025-04-30T06:35:53.763ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3818, tv_nsec: 549935995 } })
100492025-04-30T06:35:53.763ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
100502025-04-30T06:35:53.763ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
100512025-04-30T06:35:53.763ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
100522025-04-30T06:35:53.763ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100532025-04-30T06:35:53.763ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100542025-04-30T06:35:53.763ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100552025-04-30T06:35:53.764ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 93 54 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .T..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63569 req_id = db761f04-4109-4d63-b518-62a1f84f4317
100562025-04-30T06:35:53.764ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37716,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63569 req_id = db761f04-4109-4d63-b518-62a1f84f4317
100572025-04-30T06:35:53.765ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
100582025-04-30T06:35:53.765ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63569 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = db761f04-4109-4d63-b518-62a1f84f4317
100592025-04-30T06:35:53.831ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
100602025-04-30T06:35:53.832ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
100612025-04-30T06:35:53.837ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
100622025-04-30T06:35:53.837ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100632025-04-30T06:35:53.837ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100642025-04-30T06:35:53.838ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100652025-04-30T06:35:53.839ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 4d d0 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c M............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58075 req_id = ccefb331-9afb-4355-8288-6f67e631d801
100662025-04-30T06:35:53.839ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 19920,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58075 req_id = ccefb331-9afb-4355-8288-6f67e631d801
100672025-04-30T06:35:53.839ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
100682025-04-30T06:35:53.839ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 19920, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58075 req_id = ccefb331-9afb-4355-8288-6f67e631d801
100692025-04-30T06:35:53.839ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
100702025-04-30T06:35:53.839ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
100712025-04-30T06:35:53.840ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
100722025-04-30T06:35:53.846ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
100732025-04-30T06:35:53.846ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
100742025-04-30T06:35:53.849ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
100752025-04-30T06:35:53.851ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2ee212f-6427-43c4-9a01-0d20258c621f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100762025-04-30T06:35:53.851ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
100772025-04-30T06:35:53.853ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7874260e-a647-475c-981c-d36db8e9da24", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100782025-04-30T06:35:53.853ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
100792025-04-30T06:35:53.857ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0edc52bb-e458-4b64-8720-3b53f2c1b12e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100802025-04-30T06:35:53.857ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
100812025-04-30T06:35:53.859ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e6c60a7-2428-4e27-9f5c-de60847a2313", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100822025-04-30T06:35:53.861ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
100832025-04-30T06:35:53.864ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f9732fd-2ce8-4304-815d-d076b1675dd5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100842025-04-30T06:35:53.864ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
100852025-04-30T06:35:53.867ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cb49dc7e-1c30-47af-b50f-5f27fa66f3ed", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100862025-04-30T06:35:53.872ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
100872025-04-30T06:35:53.886ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6316aea-18b6-4b5a-af4b-86366a8a485e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100882025-04-30T06:35:53.886ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
100892025-04-30T06:35:53.892ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "678344bf-d5cd-49a5-8fc7-b351897f11ef", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100902025-04-30T06:35:53.892ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
100912025-04-30T06:35:53.893ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85140966-d2de-47e5-8eba-73152cc6e3f1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100922025-04-30T06:35:53.893ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
100932025-04-30T06:35:53.895ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a4e2aea-b42c-4836-9b76-9c207ae1675f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100942025-04-30T06:35:53.895ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
100952025-04-30T06:35:53.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eadee28e-5823-479e-b92e-2fada6938b7b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100962025-04-30T06:35:53.904ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
100972025-04-30T06:35:53.904ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
100982025-04-30T06:35:53.907ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9b26bc3-5976-4e27-bb06-27bebf1e92cd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
100992025-04-30T06:35:53.908ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
101002025-04-30T06:35:53.909ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17c263f6-e5b3-4932-81a3-9f7f683eb41b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101012025-04-30T06:35:53.909ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
101022025-04-30T06:35:53.911ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd3281bf-8ef3-47c4-8fc7-0aa08c753f6f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101032025-04-30T06:35:53.911ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
101042025-04-30T06:35:53.913ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "441cd6d4-9b6d-43b1-a06f-0ba9e668050a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101052025-04-30T06:35:53.913ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
101062025-04-30T06:35:53.915ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c23563fb-1aa1-412d-864b-a7ce41295229", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101072025-04-30T06:35:53.915ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
101082025-04-30T06:35:53.917ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eed386b5-9b03-4338-9291-5693e9a298f1", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101092025-04-30T06:35:53.917ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
101102025-04-30T06:35:53.919ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5563f71e-c72b-4e48-81d7-a80f8f99605e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101112025-04-30T06:35:53.919ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
101122025-04-30T06:35:53.921ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8cac4e18-daff-4dd6-8deb-968f45627112", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101132025-04-30T06:35:53.921ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
101142025-04-30T06:35:53.922ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c98ba2a7-6ca6-4a7a-b567-199ca0810581", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101152025-04-30T06:35:53.923ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
101162025-04-30T06:35:53.923ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dcfc3542-cc16-46d4-8e9b-dc7bc994b1de", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101172025-04-30T06:35:53.924ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
101182025-04-30T06:35:53.925ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8a88f9f-34be-4943-8237-21ed19983e62", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101192025-04-30T06:35:53.925ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
101202025-04-30T06:35:53.926ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed91bbbb-aa29-4824-a27b-4a4324eb0370", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101212025-04-30T06:35:53.926ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
101222025-04-30T06:35:53.927ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a115d5e-6839-4c69-be49-fb0f9d933b62", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101232025-04-30T06:35:53.927ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
101242025-04-30T06:35:53.929ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bcf9d0be-a03d-4a0c-bce7-1422391e52ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101252025-04-30T06:35:53.929ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
101262025-04-30T06:35:53.931ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b6cbf7a-624b-48fd-82db-3298320d96b8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101272025-04-30T06:35:53.931ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
101282025-04-30T06:35:53.933ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2592cc5-1c00-4168-b37c-b01485e18c1d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101292025-04-30T06:35:53.933ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
101302025-04-30T06:35:53.935ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5d78d8d8-f47b-4c04-8799-2b2599274456", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101312025-04-30T06:35:53.935ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
101322025-04-30T06:35:53.938ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "76061c4c-f422-484d-926f-65d3935c4117", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101332025-04-30T06:35:53.938ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
101342025-04-30T06:35:53.941ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "789e3d37-dd98-42d1-adfa-1a89a2209713", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101352025-04-30T06:35:53.941ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
101362025-04-30T06:35:53.942ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76e04449-e6e2-4aac-bb97-e87e2c7d064b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101372025-04-30T06:35:53.943ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
101382025-04-30T06:35:53.944ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf6fd456-96b6-40dd-b670-cb125ec17978", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101392025-04-30T06:35:53.944ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
101402025-04-30T06:35:53.946ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e633f45-4653-4deb-822f-16f85549a8a7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101412025-04-30T06:35:53.946ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
101422025-04-30T06:35:53.948ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "442c5838-9ba8-478e-98a3-e31217d7880d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101432025-04-30T06:35:53.948ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
101442025-04-30T06:35:53.950ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd907163-ba4c-4175-b397-6e4e6f22a41b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101452025-04-30T06:35:53.951ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
101462025-04-30T06:35:53.953ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6f5b3fd0-f372-4fa2-91d5-d50e8ccde145", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101472025-04-30T06:35:53.953ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
101482025-04-30T06:35:53.954ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "051118c6-d710-4609-a0ad-2b0512848690", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101492025-04-30T06:35:53.955ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
101502025-04-30T06:35:53.956ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
101512025-04-30T06:35:53.957ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101522025-04-30T06:35:53.957ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101532025-04-30T06:35:53.957ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101542025-04-30T06:35:53.963ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101552025-04-30T06:35:53.965ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6e4e0a8-b606-48fe-b0a0-97febc442c8c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101562025-04-30T06:35:53.965ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
101572025-04-30T06:35:53.966ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101582025-04-30T06:35:53.979ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
101592025-04-30T06:35:53.979ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101602025-04-30T06:35:53.979ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101612025-04-30T06:35:53.980ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101622025-04-30T06:35:53.995ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
101632025-04-30T06:35:53.995ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
101642025-04-30T06:35:53.996ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
101652025-04-30T06:35:54.001ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b97e78ad-5b3b-4cff-813b-bfa86ae8e09a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101662025-04-30T06:35:54.001ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
101672025-04-30T06:35:54.008ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc84cc01-c8d0-488c-b365-1444c4f9213a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101682025-04-30T06:35:54.008ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
101692025-04-30T06:35:54.013ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8d229af-f9e6-4302-a7f4-327e4a2f7f1c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101702025-04-30T06:35:54.013ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
101712025-04-30T06:35:54.034ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7cda9737-e50c-42d9-b71e-02f2808e6895", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101722025-04-30T06:35:54.035ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
101732025-04-30T06:35:54.047ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55251ff4-5d53-4462-acff-b7f796855872", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101742025-04-30T06:35:54.048ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
101752025-04-30T06:35:54.051ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "27ea4311-c3f8-46f9-92f1-611eeabb1b86", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101762025-04-30T06:35:54.051ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
101772025-04-30T06:35:54.053ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2b2ea921-1e66-457c-88bf-8add4247949b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
101782025-04-30T06:35:54.054ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101792025-04-30T06:35:54.054ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101802025-04-30T06:35:54.054ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101812025-04-30T06:35:54.054ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101822025-04-30T06:35:54.055ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101832025-04-30T06:35:54.062ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101842025-04-30T06:35:54.067ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101852025-04-30T06:35:54.067ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
101862025-04-30T06:35:54.067ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
101872025-04-30T06:35:54.067ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101882025-04-30T06:35:54.067ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101892025-04-30T06:35:54.068ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101902025-04-30T06:35:54.072ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
101912025-04-30T06:35:54.079ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
101922025-04-30T06:35:54.079ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
101932025-04-30T06:35:54.079ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
101942025-04-30T06:35:54.079ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
101952025-04-30T06:35:54.172ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
101962025-04-30T06:35:54.172ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
101972025-04-30T06:35:54.173ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
101982025-04-30T06:35:54.173ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:58129 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
101992025-04-30T06:35:54.174ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:58129 req_id = f2003fba-055d-41a2-bb23-963d80baae17 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
102002025-04-30T06:35:54.175ZINFOtest_metrics (dropshot): request completed latency_us = 543 local_addr = [::1]:56237 method = GET remote_addr = [::1]:58129 req_id = f2003fba-055d-41a2-bb23-963d80baae17 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
102012025-04-30T06:35:54.175ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2003fba-055d-41a2-bb23-963d80baae17", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102022025-04-30T06:35:54.179ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
102032025-04-30T06:35:54.179ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
102042025-04-30T06:35:54.179ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
102052025-04-30T06:35:54.181ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
102062025-04-30T06:35:54.181ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102072025-04-30T06:35:54.181ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102082025-04-30T06:35:54.181ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102092025-04-30T06:35:54.182ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102102025-04-30T06:35:54.185ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102112025-04-30T06:35:54.191ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
102122025-04-30T06:35:54.192ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102132025-04-30T06:35:54.192ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102142025-04-30T06:35:54.193ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102152025-04-30T06:35:54.224ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102162025-04-30T06:35:54.224ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102172025-04-30T06:35:54.224ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102182025-04-30T06:35:54.224ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102192025-04-30T06:35:54.226ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102202025-04-30T06:35:54.232ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102212025-04-30T06:35:54.249ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102222025-04-30T06:35:54.249ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102232025-04-30T06:35:54.249ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102242025-04-30T06:35:54.251ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102252025-04-30T06:35:54.261ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102262025-04-30T06:35:54.261ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102272025-04-30T06:35:54.262ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102282025-04-30T06:35:54.265ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
102292025-04-30T06:35:54.266ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
102302025-04-30T06:35:54.266ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 532.394912ms iteration = 35 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
102312025-04-30T06:35:54.266ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 36 reason = Signaled
102322025-04-30T06:35:54.266ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
102332025-04-30T06:35:54.268ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102342025-04-30T06:35:54.268ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102352025-04-30T06:35:54.268ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102362025-04-30T06:35:54.272ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102372025-04-30T06:35:54.280ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102382025-04-30T06:35:54.285ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
102392025-04-30T06:35:54.286ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
102402025-04-30T06:35:54.293ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
102412025-04-30T06:35:54.295ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
102422025-04-30T06:35:54.300ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
102432025-04-30T06:35:54.300ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
102442025-04-30T06:35:54.304ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 13 6a 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .j..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61005 req_id = 4e0d6159-16c4-48f9-a5c6-0b24a28ecbb5
102452025-04-30T06:35:54.304ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4970,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61005 req_id = 4e0d6159-16c4-48f9-a5c6-0b24a28ecbb5
102462025-04-30T06:35:54.305ZDEBGtest_metrics (store): query key key = _mgs._tcp
102472025-04-30T06:35:54.306ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
102482025-04-30T06:35:54.306ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61005 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 4e0d6159-16c4-48f9-a5c6-0b24a28ecbb5
102492025-04-30T06:35:54.307ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3819, tv_nsec: 105729607 } })
102502025-04-30T06:35:54.307ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
102512025-04-30T06:35:54.312ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: d6 ca 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:65128 req_id = d71f70f3-8c4f-4bea-8e41-d0efeea70f11
102522025-04-30T06:35:54.312ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 54986,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65128 req_id = d71f70f3-8c4f-4bea-8e41-d0efeea70f11
102532025-04-30T06:35:54.312ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
102542025-04-30T06:35:54.313ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:65128 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d71f70f3-8c4f-4bea-8e41-d0efeea70f11
102552025-04-30T06:35:54.390ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
102562025-04-30T06:35:54.398ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 18 8c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:64142 req_id = d50fa676-2e34-43b1-9e89-4b355c76361f
102572025-04-30T06:35:54.398ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6284,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64142 req_id = d50fa676-2e34-43b1-9e89-4b355c76361f
102582025-04-30T06:35:54.398ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
102592025-04-30T06:35:54.399ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 6284, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:64142 req_id = d50fa676-2e34-43b1-9e89-4b355c76361f
102602025-04-30T06:35:54.399ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
102612025-04-30T06:35:54.400ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
102622025-04-30T06:35:54.400ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
102632025-04-30T06:35:54.410ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
102642025-04-30T06:35:54.415ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "54d1494e-fcd3-4c95-a449-0410805e49c3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102652025-04-30T06:35:54.415ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
102662025-04-30T06:35:54.458ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "759351db-78ad-4408-b894-1171207f51eb", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102672025-04-30T06:35:54.459ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
102682025-04-30T06:35:54.466ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
102692025-04-30T06:35:54.466ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
102702025-04-30T06:35:54.466ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102712025-04-30T06:35:54.466ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102722025-04-30T06:35:54.467ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102732025-04-30T06:35:54.476ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
102742025-04-30T06:35:54.504ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
102752025-04-30T06:35:54.504ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102762025-04-30T06:35:54.504ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102772025-04-30T06:35:54.504ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102782025-04-30T06:35:54.507ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
102792025-04-30T06:35:54.507ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
102802025-04-30T06:35:54.508ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
102812025-04-30T06:35:54.512ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f96fc8c0-a4b4-42a7-b9a0-15b4d797f895", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102822025-04-30T06:35:54.513ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
102832025-04-30T06:35:54.515ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102842025-04-30T06:35:54.518ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102852025-04-30T06:35:54.524ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102862025-04-30T06:35:54.525ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102872025-04-30T06:35:54.525ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102882025-04-30T06:35:54.525ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
102892025-04-30T06:35:54.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6159c9d-949f-409c-b696-ebcb3d5ea5c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102902025-04-30T06:35:54.530ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
102912025-04-30T06:35:54.537ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b64f52c-7978-412d-9d08-51cae8752931", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102922025-04-30T06:35:54.537ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
102932025-04-30T06:35:54.544ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6a2e7aaf-c1cb-4d70-9f23-2b6528517275", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102942025-04-30T06:35:54.544ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
102952025-04-30T06:35:54.550ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
102962025-04-30T06:35:54.552ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c589537a-618c-4c21-aff5-0a516cd70176", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102972025-04-30T06:35:54.552ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
102982025-04-30T06:35:54.555ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8fa4231c-efd3-422e-95fb-42c0f4676aa4", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
102992025-04-30T06:35:54.555ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
103002025-04-30T06:35:54.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "634934c5-9f01-485d-8fa2-c683e2427c66", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103012025-04-30T06:35:54.556ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
103022025-04-30T06:35:54.557ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7d97088-4a76-4daa-aec3-b20026ce8dda", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103032025-04-30T06:35:54.557ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
103042025-04-30T06:35:54.559ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4fb80f6-ca11-49dc-97e6-284340f5f135", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103052025-04-30T06:35:54.559ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
103062025-04-30T06:35:54.562ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "840e1675-cff4-4aaa-a441-7312f302f386", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103072025-04-30T06:35:54.563ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
103082025-04-30T06:35:54.564ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b836825-dbe1-4eb4-82cc-9ac7ff861762", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103092025-04-30T06:35:54.565ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
103102025-04-30T06:35:54.567ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "432607fe-36f7-4c82-bcd1-1df20fa94e64", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103112025-04-30T06:35:54.567ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
103122025-04-30T06:35:54.568ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103132025-04-30T06:35:54.568ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103142025-04-30T06:35:54.568ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
103152025-04-30T06:35:54.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
103162025-04-30T06:35:54.573ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e74c1c22-d15d-447b-801f-d9a3dd1fdea5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103172025-04-30T06:35:54.573ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
103182025-04-30T06:35:54.574ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
103192025-04-30T06:35:54.574ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
103202025-04-30T06:35:54.575ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "437a9d90-2253-4504-8793-10252b0fd71f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103212025-04-30T06:35:54.575ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
103222025-04-30T06:35:54.581ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
103232025-04-30T06:35:54.582ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
103242025-04-30T06:35:54.582ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d1db31c6-8506-4d53-96d6-1a07bfb8eb30", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103252025-04-30T06:35:54.582ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
103262025-04-30T06:35:54.586ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "203ea4a3-75be-455e-a7e7-bfb784dc6423", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103272025-04-30T06:35:54.586ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
103282025-04-30T06:35:54.591ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d4acaf8-8ae2-4236-a3d1-e0a24a06ddf5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103292025-04-30T06:35:54.592ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
103302025-04-30T06:35:54.593ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98b00d8e-f658-4a91-ae65-728b41903946", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103312025-04-30T06:35:54.594ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
103322025-04-30T06:35:54.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9612f03-9cd2-47e9-a404-c1e9eafe657b", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103332025-04-30T06:35:54.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
103342025-04-30T06:35:54.597ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7366c25d-1bb0-4ae0-93cf-9b06a24314b6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103352025-04-30T06:35:54.597ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
103362025-04-30T06:35:54.599ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "55290612-e2fe-40c4-9309-38fc0b49e148", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103372025-04-30T06:35:54.599ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
103382025-04-30T06:35:54.601ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
103392025-04-30T06:35:54.601ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103402025-04-30T06:35:54.601ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103412025-04-30T06:35:54.601ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103422025-04-30T06:35:54.602ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103432025-04-30T06:35:54.604ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30467246-907a-46ce-ae57-0ebbb0094dd5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103442025-04-30T06:35:54.604ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
103452025-04-30T06:35:54.605ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
103462025-04-30T06:35:54.613ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
103472025-04-30T06:35:54.613ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103482025-04-30T06:35:54.614ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103492025-04-30T06:35:54.614ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103502025-04-30T06:35:54.620ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
103512025-04-30T06:35:54.620ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
103522025-04-30T06:35:54.621ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
103532025-04-30T06:35:54.623ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "50e0e2c1-e43f-4049-a646-b002dfc996ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103542025-04-30T06:35:54.623ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
103552025-04-30T06:35:54.626ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "841268e4-ea40-46a9-a197-da2888b37d8f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103562025-04-30T06:35:54.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
103572025-04-30T06:35:54.631ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47134b16-6ddb-4ed5-94e2-43f499c759b0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103582025-04-30T06:35:54.631ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
103592025-04-30T06:35:54.634ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7255e9ca-af73-469e-9488-15f143e5b6b3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103602025-04-30T06:35:54.634ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
103612025-04-30T06:35:54.639ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdfb2be3-7443-4564-986b-0e3fd4959ce9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103622025-04-30T06:35:54.639ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
103632025-04-30T06:35:54.642ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3a5680b-9e53-4640-ab9a-5c0d59e05ee0", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103642025-04-30T06:35:54.642ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
103652025-04-30T06:35:54.644ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd8e8692-7bed-40bd-bbde-84453c91df97", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103662025-04-30T06:35:54.644ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
103672025-04-30T06:35:54.657ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c71f1e2-121e-4cf0-8c2e-c63d5dfeafe7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103682025-04-30T06:35:54.657ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
103692025-04-30T06:35:54.660ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac1a59bb-b920-4da0-acf7-6e589f15df62", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103702025-04-30T06:35:54.660ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
103712025-04-30T06:35:54.666ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
103722025-04-30T06:35:54.666ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "52ca98c2-dc31-4c18-b2c2-7a9b94dfd0ca", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103732025-04-30T06:35:54.667ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
103742025-04-30T06:35:54.669ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
103752025-04-30T06:35:54.669ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2696426d-2884-46d0-9743-fd6e5d6acfe9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103762025-04-30T06:35:54.670ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
103772025-04-30T06:35:54.673ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a61edd4e-233c-428a-b9ec-b91bf52abd04", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103782025-04-30T06:35:54.673ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
103792025-04-30T06:35:54.677ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "024f2c7a-94cb-4c93-86ec-7941d2d00b17", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103802025-04-30T06:35:54.677ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
103812025-04-30T06:35:54.679ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83abf9e5-a543-4042-9152-08e31fdcd6c2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103822025-04-30T06:35:54.679ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
103832025-04-30T06:35:54.681ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d0bfdbd-38b2-436c-889f-a39ca2424073", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103842025-04-30T06:35:54.681ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
103852025-04-30T06:35:54.687ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "290296a0-27fb-4d8b-a593-ab9c20820065", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103862025-04-30T06:35:54.687ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
103872025-04-30T06:35:54.689ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97c4605b-65ab-4711-a3ce-9971550021d8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103882025-04-30T06:35:54.690ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
103892025-04-30T06:35:54.691ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ac81c6d-f8ff-426b-8783-80bd11e89678", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103902025-04-30T06:35:54.691ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
103912025-04-30T06:35:54.692ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8eac52c3-b29a-4530-9c6d-124ad77da818", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103922025-04-30T06:35:54.693ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
103932025-04-30T06:35:54.703ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2720b111-761b-4846-b66f-67adfbe61363", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103942025-04-30T06:35:54.703ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
103952025-04-30T06:35:54.714ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7651dddb-7c56-4bfe-920e-417ee7551e47", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:53 GMT"} })
103962025-04-30T06:35:54.715ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
103972025-04-30T06:35:54.715ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
103982025-04-30T06:35:54.715ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
103992025-04-30T06:35:54.715ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104002025-04-30T06:35:54.717ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104012025-04-30T06:35:54.719ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104022025-04-30T06:35:54.719ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104032025-04-30T06:35:54.719ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104042025-04-30T06:35:54.719ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104052025-04-30T06:35:54.720ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104062025-04-30T06:35:54.722ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104072025-04-30T06:35:54.726ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
104082025-04-30T06:35:54.726ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
104092025-04-30T06:35:54.726ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104102025-04-30T06:35:54.726ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104112025-04-30T06:35:54.726ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104122025-04-30T06:35:54.728ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104132025-04-30T06:35:54.730ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104142025-04-30T06:35:54.736ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104152025-04-30T06:35:54.736ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104162025-04-30T06:35:54.736ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104172025-04-30T06:35:54.737ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104182025-04-30T06:35:54.739ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104192025-04-30T06:35:54.768ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
104202025-04-30T06:35:54.769ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104212025-04-30T06:35:54.769ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104222025-04-30T06:35:54.771ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104232025-04-30T06:35:54.774ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104242025-04-30T06:35:54.774ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104252025-04-30T06:35:54.775ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104262025-04-30T06:35:54.889ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
104272025-04-30T06:35:54.889ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
104282025-04-30T06:35:54.890ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:54497 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
104292025-04-30T06:35:54.891ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:54497 req_id = 9dbf1118-f363-40f9-b8c3-7ed20a63a9ff sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
104302025-04-30T06:35:54.891ZINFOtest_metrics (dropshot): request completed latency_us = 590 local_addr = [::1]:56237 method = GET remote_addr = [::1]:54497 req_id = 9dbf1118-f363-40f9-b8c3-7ed20a63a9ff response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
104312025-04-30T06:35:54.893ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dbf1118-f363-40f9-b8c3-7ed20a63a9ff", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
104322025-04-30T06:35:54.895ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
104332025-04-30T06:35:54.895ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
104342025-04-30T06:35:54.895ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
104352025-04-30T06:35:54.897ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104362025-04-30T06:35:54.897ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104372025-04-30T06:35:54.897ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104382025-04-30T06:35:54.897ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104392025-04-30T06:35:54.898ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104402025-04-30T06:35:54.906ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104412025-04-30T06:35:54.911ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104422025-04-30T06:35:54.911ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104432025-04-30T06:35:54.911ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104442025-04-30T06:35:54.911ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104452025-04-30T06:35:54.981ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
104462025-04-30T06:35:54.989ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
104472025-04-30T06:35:54.990ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
104482025-04-30T06:35:54.990ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 723.67755ms iteration = 36 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
104492025-04-30T06:35:54.990ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 37 reason = Signaled
104502025-04-30T06:35:54.990ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
104512025-04-30T06:35:54.990ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104522025-04-30T06:35:54.990ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104532025-04-30T06:35:54.990ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104542025-04-30T06:35:54.991ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104552025-04-30T06:35:54.998ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104562025-04-30T06:35:55.015ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
104572025-04-30T06:35:55.016ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
104582025-04-30T06:35:55.016ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
104592025-04-30T06:35:55.016ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
104602025-04-30T06:35:55.021ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
104612025-04-30T06:35:55.021ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
104622025-04-30T06:35:55.022ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 14 2e 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61267 req_id = 25e00214-69e4-4e3b-98c3-3d0cb3f40265
104632025-04-30T06:35:55.022ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5166,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61267 req_id = 25e00214-69e4-4e3b-98c3-3d0cb3f40265
104642025-04-30T06:35:55.023ZDEBGtest_metrics (store): query key key = _mgs._tcp
104652025-04-30T06:35:55.023ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104662025-04-30T06:35:55.023ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61267 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 25e00214-69e4-4e3b-98c3-3d0cb3f40265
104672025-04-30T06:35:55.024ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3819, tv_nsec: 822994350 } })
104682025-04-30T06:35:55.024ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
104692025-04-30T06:35:55.025ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 37 e6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 7...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63592 req_id = d2ad5bee-9a3d-4c42-b3fb-699638ee356c
104702025-04-30T06:35:55.025ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 14310,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63592 req_id = d2ad5bee-9a3d-4c42-b3fb-699638ee356c
104712025-04-30T06:35:55.025ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
104722025-04-30T06:35:55.025ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63592 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d2ad5bee-9a3d-4c42-b3fb-699638ee356c
104732025-04-30T06:35:55.127ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
104742025-04-30T06:35:55.142ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
104752025-04-30T06:35:55.142ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
104762025-04-30T06:35:55.142ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104772025-04-30T06:35:55.142ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104782025-04-30T06:35:55.143ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104792025-04-30T06:35:55.144ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 78 20 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c x ..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57209 req_id = d0ba365c-a6bc-4f40-a217-1785f7c73b67
104802025-04-30T06:35:55.144ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30752,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57209 req_id = d0ba365c-a6bc-4f40-a217-1785f7c73b67
104812025-04-30T06:35:55.145ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
104822025-04-30T06:35:55.145ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30752, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57209 req_id = d0ba365c-a6bc-4f40-a217-1785f7c73b67
104832025-04-30T06:35:55.145ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
104842025-04-30T06:35:55.145ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
104852025-04-30T06:35:55.145ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
104862025-04-30T06:35:55.146ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
104872025-04-30T06:35:55.153ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
104882025-04-30T06:35:55.154ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104892025-04-30T06:35:55.154ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104902025-04-30T06:35:55.155ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104912025-04-30T06:35:55.159ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
104922025-04-30T06:35:55.159ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
104932025-04-30T06:35:55.163ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
104942025-04-30T06:35:55.165ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b9b67ed-28a9-4842-99a3-c2fb6341ca3e", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
104952025-04-30T06:35:55.166ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
104962025-04-30T06:35:55.169ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd93a5d1-320c-4b32-bea2-b8c1e1e9a31c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
104972025-04-30T06:35:55.169ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
104982025-04-30T06:35:55.173ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f53c889-e7e9-4596-ac84-251432608e5f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
104992025-04-30T06:35:55.173ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
105002025-04-30T06:35:55.176ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d4bd14c-4a0b-401c-9b3f-0f5527479fe0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105012025-04-30T06:35:55.177ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
105022025-04-30T06:35:55.184ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f8d9dbc-1644-4d34-bbd9-04fc87dee593", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105032025-04-30T06:35:55.184ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
105042025-04-30T06:35:55.187ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b50790d-5d9c-458b-9956-4e665820875c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105052025-04-30T06:35:55.187ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
105062025-04-30T06:35:55.191ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a67546cf-6071-4603-997f-6eb3085140f7", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105072025-04-30T06:35:55.192ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
105082025-04-30T06:35:55.206ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fa410833-dd1f-48c6-9dde-eae92d0081e2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105092025-04-30T06:35:55.206ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
105102025-04-30T06:35:55.207ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
105112025-04-30T06:35:55.208ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "424178b1-a258-42fe-aeff-82d51fe8b701", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105122025-04-30T06:35:55.209ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
105132025-04-30T06:35:55.210ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb56435f-3e56-4b2b-bc9a-e6ec0c6213b5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105142025-04-30T06:35:55.210ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
105152025-04-30T06:35:55.211ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76085c4b-54b2-433e-a7f0-7949336a1953", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105162025-04-30T06:35:55.212ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
105172025-04-30T06:35:55.213ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "acc84e5b-f9f9-4147-8a04-d599fb572338", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105182025-04-30T06:35:55.213ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
105192025-04-30T06:35:55.214ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "283b4525-9d32-4d46-aaf4-4a0e727c91a8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105202025-04-30T06:35:55.214ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
105212025-04-30T06:35:55.216ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "960d7b59-9de8-46b8-a3db-323aadae516c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105222025-04-30T06:35:55.217ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
105232025-04-30T06:35:55.220ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00746ee9-7cbb-4aec-988c-ed837c36ba3e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105242025-04-30T06:35:55.220ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
105252025-04-30T06:35:55.223ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "48d41e1d-001b-4e99-8e59-1aebc3a58de6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105262025-04-30T06:35:55.224ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
105272025-04-30T06:35:55.228ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "943addc3-c22d-4b12-b0dd-ad6aca0870f9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105282025-04-30T06:35:55.228ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
105292025-04-30T06:35:55.235ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "06665515-f333-4db4-8209-f908eda0cff1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105302025-04-30T06:35:55.235ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
105312025-04-30T06:35:55.239ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fc07c5a7-fc13-41ec-b101-804a3939a773", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105322025-04-30T06:35:55.239ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
105332025-04-30T06:35:55.243ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b37ed928-c3eb-47cf-89cd-310e55dae8a1", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105342025-04-30T06:35:55.244ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
105352025-04-30T06:35:55.251ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bdb499e5-0fcf-45d8-aeb4-b271ec7291b9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105362025-04-30T06:35:55.252ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
105372025-04-30T06:35:55.256ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85bc0fed-4110-4976-888b-3c770e5ae71f", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105382025-04-30T06:35:55.256ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
105392025-04-30T06:35:55.259ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105402025-04-30T06:35:55.259ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105412025-04-30T06:35:55.259ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105422025-04-30T06:35:55.259ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105432025-04-30T06:35:55.261ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105442025-04-30T06:35:55.261ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5417155a-e2a4-4e17-b908-0de2c11f14a6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105452025-04-30T06:35:55.262ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
105462025-04-30T06:35:55.268ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105472025-04-30T06:35:55.282ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105482025-04-30T06:35:55.282ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105492025-04-30T06:35:55.282ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105502025-04-30T06:35:55.283ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105512025-04-30T06:35:55.287ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "afe9494d-fa96-4d2f-99f4-69f7f0fa006b", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105522025-04-30T06:35:55.288ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
105532025-04-30T06:35:55.289ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105542025-04-30T06:35:55.289ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105552025-04-30T06:35:55.291ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105562025-04-30T06:35:55.304ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
105572025-04-30T06:35:55.305ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105582025-04-30T06:35:55.307ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 067b6688-0052-44b4-997a-018b283e9044 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105592025-04-30T06:35:55.308ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 067b6688-0052-44b4-997a-018b283e9044 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105602025-04-30T06:35:55.309ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 067b6688-0052-44b4-997a-018b283e9044 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105612025-04-30T06:35:55.318ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 067b6688-0052-44b4-997a-018b283e9044 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105622025-04-30T06:35:55.325ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae2e7628-d680-4965-8393-f7336bdf93aa", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105632025-04-30T06:35:55.326ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
105642025-04-30T06:35:55.326ZINFOtest_metrics (dropshot_internal): request completed latency_us = 20359 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 067b6688-0052-44b4-997a-018b283e9044 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
105652025-04-30T06:35:55.328ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "067b6688-0052-44b4-997a-018b283e9044", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105662025-04-30T06:35:55.328ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
105672025-04-30T06:35:55.330ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = c78f9609-ba3f-4517-b650-c815d3abbc10 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
105682025-04-30T06:35:55.332ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = c78f9609-ba3f-4517-b650-c815d3abbc10 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
105692025-04-30T06:35:55.332ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = c78f9609-ba3f-4517-b650-c815d3abbc10 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
105702025-04-30T06:35:55.333ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = c78f9609-ba3f-4517-b650-c815d3abbc10 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
105712025-04-30T06:35:55.338ZINFOtest_metrics (dropshot_internal): request completed latency_us = 7640 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = c78f9609-ba3f-4517-b650-c815d3abbc10 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
105722025-04-30T06:35:55.339ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "c78f9609-ba3f-4517-b650-c815d3abbc10", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105732025-04-30T06:35:55.340ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
105742025-04-30T06:35:55.340ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
105752025-04-30T06:35:55.340ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
105762025-04-30T06:35:55.383ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
105772025-04-30T06:35:55.394ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31acb07e-cca3-47dc-b6b5-95c32a8761a8", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105782025-04-30T06:35:55.394ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
105792025-04-30T06:35:55.423ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6dd877e6-508e-48a8-b099-a28912c6d913", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105802025-04-30T06:35:55.423ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
105812025-04-30T06:35:55.429ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b9afe55-ba5c-428d-b0b0-a8151d7aee78", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105822025-04-30T06:35:55.429ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
105832025-04-30T06:35:55.444ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
105842025-04-30T06:35:55.445ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
105852025-04-30T06:35:55.445ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105862025-04-30T06:35:55.445ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105872025-04-30T06:35:55.448ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105882025-04-30T06:35:55.455ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
105892025-04-30T06:35:55.468ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
105902025-04-30T06:35:55.468ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105912025-04-30T06:35:55.468ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105922025-04-30T06:35:55.469ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105932025-04-30T06:35:55.473ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "db595610-a6b3-4991-8ab3-eabe056bfb5a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105942025-04-30T06:35:55.475ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
105952025-04-30T06:35:55.475ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
105962025-04-30T06:35:55.475ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
105972025-04-30T06:35:55.479ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
105982025-04-30T06:35:55.481ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bb6e80d-9b24-481a-b43c-025ba9ff2b92", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
105992025-04-30T06:35:55.481ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
106002025-04-30T06:35:55.483ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3a6ede2d-3146-4e0b-a58c-2825a8f1facd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106012025-04-30T06:35:55.483ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
106022025-04-30T06:35:55.484ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8bd3416-b97d-4f9d-a797-bb6163cb9a28", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106032025-04-30T06:35:55.484ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
106042025-04-30T06:35:55.487ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e21fece-9fb5-44ba-8d7c-ea4ae5ad0cfa", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106052025-04-30T06:35:55.488ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
106062025-04-30T06:35:55.489ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e678b8b2-04ef-4159-8b31-2c85f69ffc6f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106072025-04-30T06:35:55.489ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
106082025-04-30T06:35:55.491ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "046670af-c2e1-4cac-aa1c-0f1ce193c511", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106092025-04-30T06:35:55.491ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
106102025-04-30T06:35:55.497ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a70ded9b-9102-45f3-a388-bd9f95ec4358", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106112025-04-30T06:35:55.497ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
106122025-04-30T06:35:55.503ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d09e1b1-41ae-48c4-aeeb-3ce30737fe2f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106132025-04-30T06:35:55.503ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
106142025-04-30T06:35:55.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80ab3f9-ff2e-4b5e-b289-ef7f632a49ae", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106152025-04-30T06:35:55.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
106162025-04-30T06:35:55.510ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
106172025-04-30T06:35:55.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106182025-04-30T06:35:55.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106192025-04-30T06:35:55.514ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106202025-04-30T06:35:55.515ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106212025-04-30T06:35:55.515ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106222025-04-30T06:35:55.515ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
106232025-04-30T06:35:55.516ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "20145c59-f0c8-40b7-b9f2-7a8bf858cc01", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106242025-04-30T06:35:55.516ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
106252025-04-30T06:35:55.521ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced673ef-a7a5-4f51-8f36-cf964ec51638", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106262025-04-30T06:35:55.522ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
106272025-04-30T06:35:55.524ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2bcb2f45-fc06-4598-b2f8-f8abf3b51d2c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106282025-04-30T06:35:55.524ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
106292025-04-30T06:35:55.525ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cca58c0e-7c2d-43a2-a12e-1e9817661f82", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106302025-04-30T06:35:55.525ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
106312025-04-30T06:35:55.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9fe00c-cc5d-4d32-b45c-4349f3655bff", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106322025-04-30T06:35:55.527ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
106332025-04-30T06:35:55.529ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fa1a3a8-1d34-4520-a781-972310a5be5a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106342025-04-30T06:35:55.529ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
106352025-04-30T06:35:55.531ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1a4baeb2-6d58-4f75-9d35-048403263bb6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106362025-04-30T06:35:55.531ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
106372025-04-30T06:35:55.531ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106382025-04-30T06:35:55.531ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106392025-04-30T06:35:55.531ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106402025-04-30T06:35:55.533ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106412025-04-30T06:35:55.544ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106422025-04-30T06:35:55.552ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
106432025-04-30T06:35:55.552ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
106442025-04-30T06:35:55.552ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106452025-04-30T06:35:55.552ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106462025-04-30T06:35:55.552ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106472025-04-30T06:35:55.553ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106482025-04-30T06:35:55.557ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106492025-04-30T06:35:55.560ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
106502025-04-30T06:35:55.560ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106512025-04-30T06:35:55.560ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106522025-04-30T06:35:55.561ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106532025-04-30T06:35:55.562ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
106542025-04-30T06:35:55.562ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106552025-04-30T06:35:55.562ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106562025-04-30T06:35:55.562ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106572025-04-30T06:35:55.562ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106582025-04-30T06:35:55.565ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106592025-04-30T06:35:55.569ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
106602025-04-30T06:35:55.569ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106612025-04-30T06:35:55.569ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106622025-04-30T06:35:55.570ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106632025-04-30T06:35:55.628ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
106642025-04-30T06:35:55.628ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
106652025-04-30T06:35:55.629ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:37448 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
106662025-04-30T06:35:55.630ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106672025-04-30T06:35:55.631ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106682025-04-30T06:35:55.632ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
106692025-04-30T06:35:55.633ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
106702025-04-30T06:35:55.633ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
106712025-04-30T06:35:55.633ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
106722025-04-30T06:35:55.634ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
106732025-04-30T06:35:55.634ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:37448 req_id = 3ed7d841-78e7-4e33-a9db-5b7e2eebebe4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106742025-04-30T06:35:55.634ZINFOtest_metrics (dropshot): request completed latency_us = 725 local_addr = [::1]:56237 method = GET remote_addr = [::1]:37448 req_id = 3ed7d841-78e7-4e33-a9db-5b7e2eebebe4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
106752025-04-30T06:35:55.635ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ed7d841-78e7-4e33-a9db-5b7e2eebebe4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:54 GMT"} })
106762025-04-30T06:35:55.638ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106772025-04-30T06:35:55.638ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
106782025-04-30T06:35:55.638ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
106792025-04-30T06:35:55.640ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106802025-04-30T06:35:55.640ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
106812025-04-30T06:35:55.640ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106822025-04-30T06:35:55.641ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106832025-04-30T06:35:55.643ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106842025-04-30T06:35:55.645ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
106852025-04-30T06:35:55.645ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
106862025-04-30T06:35:55.648ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
106872025-04-30T06:35:55.663ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
106882025-04-30T06:35:55.663ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
106892025-04-30T06:35:55.663ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
106902025-04-30T06:35:55.664ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
106912025-04-30T06:35:55.672ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
106922025-04-30T06:35:55.672ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
106932025-04-30T06:35:55.739ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
106942025-04-30T06:35:55.837ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
106952025-04-30T06:35:55.837ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
106962025-04-30T06:35:55.838ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 847.769325ms iteration = 37 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
106972025-04-30T06:35:55.838ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 38 reason = Signaled
106982025-04-30T06:35:55.838ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
106992025-04-30T06:35:55.838ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107002025-04-30T06:35:55.838ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107012025-04-30T06:35:55.838ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107022025-04-30T06:35:55.844ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107032025-04-30T06:35:55.856ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
107042025-04-30T06:35:55.860ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107052025-04-30T06:35:55.864ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
107062025-04-30T06:35:55.864ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
107072025-04-30T06:35:55.864ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
107082025-04-30T06:35:55.864ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
107092025-04-30T06:35:55.872ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
107102025-04-30T06:35:55.872ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
107112025-04-30T06:35:55.873ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 92 35 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .5..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:60542 req_id = 8d0edb8b-b97e-4aa3-8a7e-70bd4a22aa27
107122025-04-30T06:35:55.873ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37429,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60542 req_id = 8d0edb8b-b97e-4aa3-8a7e-70bd4a22aa27
107132025-04-30T06:35:55.873ZDEBGtest_metrics (store): query key key = _mgs._tcp
107142025-04-30T06:35:55.873ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
107152025-04-30T06:35:55.898ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:60542 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8d0edb8b-b97e-4aa3-8a7e-70bd4a22aa27
107162025-04-30T06:35:55.901ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3820, tv_nsec: 700415995 } })
107172025-04-30T06:35:55.902ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
107182025-04-30T06:35:55.903ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a9 ea 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56349 req_id = fa213619-1fba-490d-92ff-b23ed8d6ee0b
107192025-04-30T06:35:55.903ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 43498,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56349 req_id = fa213619-1fba-490d-92ff-b23ed8d6ee0b
107202025-04-30T06:35:55.904ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
107212025-04-30T06:35:55.904ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56349 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fa213619-1fba-490d-92ff-b23ed8d6ee0b
107222025-04-30T06:35:55.968ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
107232025-04-30T06:35:55.969ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107242025-04-30T06:35:55.969ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107252025-04-30T06:35:55.969ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107262025-04-30T06:35:55.969ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
107272025-04-30T06:35:55.970ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107282025-04-30T06:35:55.970ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 78 1d 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c x............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58864 req_id = 48ef84a1-8842-4aa2-8cde-632141b543d3
107292025-04-30T06:35:55.970ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30749,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58864 req_id = 48ef84a1-8842-4aa2-8cde-632141b543d3
107302025-04-30T06:35:55.970ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
107312025-04-30T06:35:55.970ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 30749, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58864 req_id = 48ef84a1-8842-4aa2-8cde-632141b543d3
107322025-04-30T06:35:55.970ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
107332025-04-30T06:35:55.970ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
107342025-04-30T06:35:55.970ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
107352025-04-30T06:35:55.972ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "71aff2be-a0b3-4088-8d7d-5b842d715a1b", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107362025-04-30T06:35:55.973ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
107372025-04-30T06:35:55.973ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107382025-04-30T06:35:55.978ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107392025-04-30T06:35:55.978ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107402025-04-30T06:35:55.978ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
107412025-04-30T06:35:55.978ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107422025-04-30T06:35:55.982ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6c3c0f6-bb36-4500-8383-ff463ca634fd", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107432025-04-30T06:35:55.982ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
107442025-04-30T06:35:55.982ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107452025-04-30T06:35:55.982ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
107462025-04-30T06:35:55.983ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107472025-04-30T06:35:55.992ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5cbcb019-620d-42ef-919c-ce582e372ee8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107482025-04-30T06:35:55.993ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
107492025-04-30T06:35:55.995ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d8cb3063-3ffd-47fb-b290-669a51bbcc5a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107502025-04-30T06:35:55.995ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
107512025-04-30T06:35:56.020ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "593eb2bb-7a9b-47a8-8910-954ae5990bfd", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107522025-04-30T06:35:56.021ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
107532025-04-30T06:35:56.025ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2aec864a-f726-4465-bd90-f9cb397eab8b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107542025-04-30T06:35:56.026ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
107552025-04-30T06:35:56.029ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c208f654-e78d-4fd5-9c5b-da3b11e49d62", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107562025-04-30T06:35:56.029ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
107572025-04-30T06:35:56.032ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba121137-56e9-474c-87b0-d65359c13711", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107582025-04-30T06:35:56.032ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
107592025-04-30T06:35:56.035ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9b9c487-ffe1-4722-8173-9f076cca93dd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107602025-04-30T06:35:56.035ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
107612025-04-30T06:35:56.039ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d49104ae-1466-44f6-a1b5-deb01f528615", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107622025-04-30T06:35:56.039ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
107632025-04-30T06:35:56.040ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7639bf43-b6cb-49cc-8918-92df640dc865", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107642025-04-30T06:35:56.041ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
107652025-04-30T06:35:56.050ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
107662025-04-30T06:35:56.051ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ae1a87b-2373-4e4c-b38c-c69523acf908", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107672025-04-30T06:35:56.052ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
107682025-04-30T06:35:56.055ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed4daa5f-a1c9-4c39-9084-8784bbc48573", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107692025-04-30T06:35:56.055ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
107702025-04-30T06:35:56.060ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbd9e12a-ecf4-4dd9-a951-6dbee50aae47", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107712025-04-30T06:35:56.060ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
107722025-04-30T06:35:56.070ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dba16144-29c2-4bcd-ab9c-5b9814fa5310", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107732025-04-30T06:35:56.070ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
107742025-04-30T06:35:56.088ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "044e524c-c355-424b-8fcd-20ef31549568", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107752025-04-30T06:35:56.089ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
107762025-04-30T06:35:56.091ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9e5daad-e09c-42a5-838a-bf29502e92ff", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107772025-04-30T06:35:56.091ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
107782025-04-30T06:35:56.094ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df0fae45-6326-41d3-bcd9-8b2cfb07f58c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107792025-04-30T06:35:56.094ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
107802025-04-30T06:35:56.096ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aab25227-8af8-4000-8924-25c76ca818fa", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107812025-04-30T06:35:56.096ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
107822025-04-30T06:35:56.097ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a61aae0b-d6ca-4c1a-9e4a-cb05b7b55e8c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107832025-04-30T06:35:56.098ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
107842025-04-30T06:35:56.099ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2a06365-fa71-4bd4-aeaa-6f0ddfe0bfdb", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107852025-04-30T06:35:56.099ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
107862025-04-30T06:35:56.100ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ed4ff2ab-fef7-469b-a4bc-23eec51a8990", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107872025-04-30T06:35:56.100ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
107882025-04-30T06:35:56.101ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
107892025-04-30T06:35:56.101ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
107902025-04-30T06:35:56.101ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107912025-04-30T06:35:56.101ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
107922025-04-30T06:35:56.102ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107932025-04-30T06:35:56.107ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
107942025-04-30T06:35:56.163ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
107952025-04-30T06:35:56.163ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
107962025-04-30T06:35:56.163ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
107972025-04-30T06:35:56.164ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
107982025-04-30T06:35:56.168ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "043b2363-35dc-48c5-ae9d-1e5887e7f6f5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
107992025-04-30T06:35:56.168ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
108002025-04-30T06:35:56.169ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108012025-04-30T06:35:56.169ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
108022025-04-30T06:35:56.172ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108032025-04-30T06:35:56.206ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2e0c56-7387-4a32-95de-a4554491ba25", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108042025-04-30T06:35:56.206ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
108052025-04-30T06:35:56.214ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ea681bfa-9243-4583-8b2d-cbd98469d481", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108062025-04-30T06:35:56.214ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
108072025-04-30T06:35:56.250ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8bbb86f-2948-49c5-ae07-14fe6de80955", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108082025-04-30T06:35:56.250ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
108092025-04-30T06:35:56.258ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "614039cc-c87e-497a-aa99-21cf70f65cd9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108102025-04-30T06:35:56.258ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
108112025-04-30T06:35:56.266ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a5a55eef-0c5d-4337-9eab-24b040f037ba", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108122025-04-30T06:35:56.267ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
108132025-04-30T06:35:56.267ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
108142025-04-30T06:35:56.282ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb1179e2-7dbf-4782-9f05-3b1184204fdd", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108152025-04-30T06:35:56.282ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
108162025-04-30T06:35:56.309ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b995a1c-f3a4-41f0-9a67-2d8ebba54d92", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108172025-04-30T06:35:56.310ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
108182025-04-30T06:35:56.311ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7a586ebd-8c9b-4e89-843f-adeed7b328fd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108192025-04-30T06:35:56.314ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
108202025-04-30T06:35:56.316ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "921ff208-ae93-4abf-b5cb-71f8509d3b88", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108212025-04-30T06:35:56.317ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
108222025-04-30T06:35:56.320ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108232025-04-30T06:35:56.320ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108242025-04-30T06:35:56.320ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108252025-04-30T06:35:56.320ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
108262025-04-30T06:35:56.322ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108272025-04-30T06:35:56.347ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108282025-04-30T06:35:56.355ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108292025-04-30T06:35:56.358ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108302025-04-30T06:35:56.360ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
108312025-04-30T06:35:56.361ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108322025-04-30T06:35:56.435ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb63ff02-34c4-4f06-9998-ddc664029d20", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108332025-04-30T06:35:56.435ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
108342025-04-30T06:35:56.446ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108352025-04-30T06:35:56.446ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
108362025-04-30T06:35:56.447ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108372025-04-30T06:35:56.491ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "29342c29-1aa8-4b4c-a39a-26b061aa46f5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108382025-04-30T06:35:56.492ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
108392025-04-30T06:35:56.510ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b44afdfb-a97f-412e-84c8-fde947ce10b6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108402025-04-30T06:35:56.510ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
108412025-04-30T06:35:56.514ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108422025-04-30T06:35:56.515ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108432025-04-30T06:35:56.538ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108442025-04-30T06:35:56.540ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108452025-04-30T06:35:56.540ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108462025-04-30T06:35:56.540ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
108472025-04-30T06:35:56.548ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a464d99a-95c2-47bc-a4f2-a2b31b4eb6bf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108482025-04-30T06:35:56.548ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
108492025-04-30T06:35:56.561ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d51a36a1-7e88-480e-98f9-3868834c04a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108502025-04-30T06:35:56.561ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
108512025-04-30T06:35:56.564ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
108522025-04-30T06:35:56.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0c0105f-68bf-46f0-a7a6-6d69e20cd8af", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108532025-04-30T06:35:56.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
108542025-04-30T06:35:56.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108552025-04-30T06:35:56.567ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108562025-04-30T06:35:56.567ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
108572025-04-30T06:35:56.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
108582025-04-30T06:35:56.570ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0cc51d39-aba6-4ec5-a32b-ac3e0bef0076", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108592025-04-30T06:35:56.570ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
108602025-04-30T06:35:56.572ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccd2d1a2-266c-4ca9-a654-a23f2c6d9f08", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108612025-04-30T06:35:56.572ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
108622025-04-30T06:35:56.573ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
108632025-04-30T06:35:56.573ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
108642025-04-30T06:35:56.575ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d01df1c-eafd-405c-a2ef-dfd9cd53fbdb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108652025-04-30T06:35:56.575ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
108662025-04-30T06:35:56.576ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
108672025-04-30T06:35:56.576ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
108682025-04-30T06:35:56.577ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ddc20cc-75ae-4ca3-9ab1-b1e0364ee0e0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108692025-04-30T06:35:56.577ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
108702025-04-30T06:35:56.578ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08b91607-3775-4352-87bf-65dec0ed7178", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108712025-04-30T06:35:56.579ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
108722025-04-30T06:35:56.579ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93f361d5-fdc5-4c43-9a8e-f26731c04667", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108732025-04-30T06:35:56.580ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
108742025-04-30T06:35:56.580ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "da2960b1-8184-4e9d-90d8-f8374659aee1", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
108752025-04-30T06:35:56.581ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108762025-04-30T06:35:56.581ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108772025-04-30T06:35:56.581ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108782025-04-30T06:35:56.581ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108792025-04-30T06:35:56.582ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108802025-04-30T06:35:56.589ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108812025-04-30T06:35:56.598ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108822025-04-30T06:35:56.598ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
108832025-04-30T06:35:56.598ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108842025-04-30T06:35:56.598ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108852025-04-30T06:35:56.598ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108862025-04-30T06:35:56.598ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108872025-04-30T06:35:56.602ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108882025-04-30T06:35:56.619ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
108892025-04-30T06:35:56.619ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
108902025-04-30T06:35:56.619ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
108912025-04-30T06:35:56.620ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
108922025-04-30T06:35:56.620ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
108932025-04-30T06:35:56.620ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
108942025-04-30T06:35:56.620ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
108952025-04-30T06:35:56.620ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
108962025-04-30T06:35:56.622ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
108972025-04-30T06:35:56.626ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
108982025-04-30T06:35:56.631ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
108992025-04-30T06:35:56.631ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109002025-04-30T06:35:56.631ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109012025-04-30T06:35:56.632ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109022025-04-30T06:35:56.732ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
109032025-04-30T06:35:56.732ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
109042025-04-30T06:35:56.739ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:45406 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
109052025-04-30T06:35:56.740ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
109062025-04-30T06:35:56.740ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:45406 req_id = 433348b7-c7bd-44f8-83e6-df3541fc290c sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
109072025-04-30T06:35:56.741ZINFOtest_metrics (dropshot): request completed latency_us = 838 local_addr = [::1]:56237 method = GET remote_addr = [::1]:45406 req_id = 433348b7-c7bd-44f8-83e6-df3541fc290c response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
109082025-04-30T06:35:56.741ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109092025-04-30T06:35:56.741ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109102025-04-30T06:35:56.742ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109112025-04-30T06:35:56.742ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "433348b7-c7bd-44f8-83e6-df3541fc290c", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:55 GMT"} })
109122025-04-30T06:35:56.745ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
109132025-04-30T06:35:56.745ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
109142025-04-30T06:35:56.745ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
109152025-04-30T06:35:56.747ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109162025-04-30T06:35:56.747ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109172025-04-30T06:35:56.747ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109182025-04-30T06:35:56.747ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109192025-04-30T06:35:56.747ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109202025-04-30T06:35:56.761ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109212025-04-30T06:35:56.764ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109222025-04-30T06:35:56.764ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109232025-04-30T06:35:56.764ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109242025-04-30T06:35:56.765ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109252025-04-30T06:35:56.880ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
109262025-04-30T06:35:56.894ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
109272025-04-30T06:35:56.894ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
109282025-04-30T06:35:56.894ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.056393885s iteration = 38 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
109292025-04-30T06:35:56.894ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 39 reason = Signaled
109302025-04-30T06:35:56.894ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
109312025-04-30T06:35:56.894ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109322025-04-30T06:35:56.894ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109332025-04-30T06:35:56.894ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109342025-04-30T06:35:56.896ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109352025-04-30T06:35:56.908ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109362025-04-30T06:35:56.918ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
109372025-04-30T06:35:56.918ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
109382025-04-30T06:35:56.918ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
109392025-04-30T06:35:56.919ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
109402025-04-30T06:35:56.927ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
109412025-04-30T06:35:56.927ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
109422025-04-30T06:35:56.928ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: be 91 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:62592 req_id = 59c58142-35b4-40df-83f7-d3228ad87eb2
109432025-04-30T06:35:56.928ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48785,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62592 req_id = 59c58142-35b4-40df-83f7-d3228ad87eb2
109442025-04-30T06:35:56.928ZDEBGtest_metrics (store): query key key = _mgs._tcp
109452025-04-30T06:35:56.928ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109462025-04-30T06:35:56.928ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62592 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 59c58142-35b4-40df-83f7-d3228ad87eb2
109472025-04-30T06:35:56.933ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109482025-04-30T06:35:56.933ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109492025-04-30T06:35:56.934ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109502025-04-30T06:35:56.934ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109512025-04-30T06:35:56.935ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109522025-04-30T06:35:56.935ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3821, tv_nsec: 734353839 } })
109532025-04-30T06:35:56.935ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
109542025-04-30T06:35:56.939ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 43 61 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Ca..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51320 req_id = 089a1b05-8e12-43ae-b023-78218b63f4d8
109552025-04-30T06:35:56.939ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17249,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51320 req_id = 089a1b05-8e12-43ae-b023-78218b63f4d8
109562025-04-30T06:35:56.939ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
109572025-04-30T06:35:56.939ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51320 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 089a1b05-8e12-43ae-b023-78218b63f4d8
109582025-04-30T06:35:57.012ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
109592025-04-30T06:35:57.013ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
109602025-04-30T06:35:57.038ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
109612025-04-30T06:35:57.038ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109622025-04-30T06:35:57.038ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109632025-04-30T06:35:57.038ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109642025-04-30T06:35:57.039ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 8d e8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49899 req_id = 48aebd40-35cb-42d7-9b6e-bb25bcc47799
109652025-04-30T06:35:57.039ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36328,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49899 req_id = 48aebd40-35cb-42d7-9b6e-bb25bcc47799
109662025-04-30T06:35:57.040ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
109672025-04-30T06:35:57.040ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 36328, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49899 req_id = 48aebd40-35cb-42d7-9b6e-bb25bcc47799
109682025-04-30T06:35:57.041ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
109692025-04-30T06:35:57.042ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
109702025-04-30T06:35:57.042ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
109712025-04-30T06:35:57.042ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109722025-04-30T06:35:57.042ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109732025-04-30T06:35:57.043ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
109742025-04-30T06:35:57.052ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8489c53-abc9-4a86-b3cf-4c6a767f006f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109752025-04-30T06:35:57.053ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
109762025-04-30T06:35:57.059ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6de50c8-b99d-476b-81b5-a6f8fd075816", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109772025-04-30T06:35:57.059ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
109782025-04-30T06:35:57.067ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80abb7c-d5ab-42cc-ba58-b4fc788e490d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109792025-04-30T06:35:57.069ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
109802025-04-30T06:35:57.107ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "677244e2-bbe3-4f0a-ac72-1f48053a4b2c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109812025-04-30T06:35:57.107ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
109822025-04-30T06:35:57.113ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1457177-4506-41ad-9b9e-88072136d8de", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109832025-04-30T06:35:57.114ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
109842025-04-30T06:35:57.115ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
109852025-04-30T06:35:57.117ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "304ebc6b-d54e-471b-9dc7-b99370ac7732", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109862025-04-30T06:35:57.117ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
109872025-04-30T06:35:57.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "152fe43f-d40e-4dfd-87d1-289ec50f4967", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109882025-04-30T06:35:57.121ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
109892025-04-30T06:35:57.146ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c80a970a-f900-44f3-8912-35fbf6312522", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109902025-04-30T06:35:57.146ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
109912025-04-30T06:35:57.160ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0ac69ade-814f-44fd-9c05-e140e8c60cfa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109922025-04-30T06:35:57.160ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
109932025-04-30T06:35:57.162ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e372a4fe-2b35-42ab-8080-9dc3236827d7", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
109942025-04-30T06:35:57.165ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
109952025-04-30T06:35:57.166ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
109962025-04-30T06:35:57.166ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
109972025-04-30T06:35:57.166ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
109982025-04-30T06:35:57.166ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
109992025-04-30T06:35:57.167ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110002025-04-30T06:35:57.169ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5991c77-16da-427d-837a-c6e45af7b345", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110012025-04-30T06:35:57.170ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
110022025-04-30T06:35:57.171ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110032025-04-30T06:35:57.187ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110042025-04-30T06:35:57.187ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110052025-04-30T06:35:57.187ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110062025-04-30T06:35:57.188ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110072025-04-30T06:35:57.188ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d45fbffd-bbe9-4327-ba67-a7b40d236e55", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110082025-04-30T06:35:57.188ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
110092025-04-30T06:35:57.191ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110102025-04-30T06:35:57.191ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110112025-04-30T06:35:57.192ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110122025-04-30T06:35:57.192ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "35756d25-10b9-4eff-8aec-7d45cbd52176", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110132025-04-30T06:35:57.193ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
110142025-04-30T06:35:57.196ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "30530134-6016-4337-b241-a7edcc66d32e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110152025-04-30T06:35:57.196ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
110162025-04-30T06:35:57.200ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3314fa26-c761-40c8-9802-6c58dd5ad5f5", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110172025-04-30T06:35:57.200ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
110182025-04-30T06:35:57.204ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f4c8da78-d96f-4e84-9ade-ab1e76dbca3a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110192025-04-30T06:35:57.204ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
110202025-04-30T06:35:57.218ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12ceae74-ef0d-44b8-90d2-27358a7e137f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110212025-04-30T06:35:57.218ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
110222025-04-30T06:35:57.222ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "007ba646-5804-43cd-964a-72b823629a16", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110232025-04-30T06:35:57.222ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
110242025-04-30T06:35:57.229ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e15d100-106a-480e-be1f-d79d8eb54335", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110252025-04-30T06:35:57.229ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
110262025-04-30T06:35:57.230ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
110272025-04-30T06:35:57.231ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84b165fb-f731-41b6-bf2c-1fd1a25be02b", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110282025-04-30T06:35:57.231ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
110292025-04-30T06:35:57.239ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d9f8e01-1771-41f2-8778-6e5577b00dce", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110302025-04-30T06:35:57.239ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
110312025-04-30T06:35:57.276ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2638a5ca-7435-4351-a043-98c010bd4322", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110322025-04-30T06:35:57.276ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
110332025-04-30T06:35:57.277ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e753d70-bb0e-4d8b-8e00-2cbdd8f0e038", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110342025-04-30T06:35:57.277ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
110352025-04-30T06:35:57.279ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26478018-4e18-419b-a212-5e0e9cc0adca", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110362025-04-30T06:35:57.281ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
110372025-04-30T06:35:57.281ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
110382025-04-30T06:35:57.281ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110392025-04-30T06:35:57.281ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110402025-04-30T06:35:57.281ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110412025-04-30T06:35:57.282ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110422025-04-30T06:35:57.288ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
110432025-04-30T06:35:57.314ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
110442025-04-30T06:35:57.314ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110452025-04-30T06:35:57.314ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
110462025-04-30T06:35:57.315ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
110472025-04-30T06:35:57.334ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
110482025-04-30T06:35:57.342ZDEBGtest_metrics (BackgroundTasks): activating background_task = nat_v4_garbage_collector iteration = 2 reason = Timeout
110492025-04-30T06:35:57.342ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database
110502025-04-30T06:35:57.342ZDEBGtest_metrics (BackgroundTasks): roles background_task = nat_v4_garbage_collector roles = RoleSet { roles: {} }
110512025-04-30T06:35:57.343ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = nat_v4_garbage_collector resource = Database result = Ok(())
110522025-04-30T06:35:57.343ZDEBGtest_metrics (BackgroundTasks): activating background_task = bfd_manager iteration = 2 reason = Timeout
110532025-04-30T06:35:57.343ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database
110542025-04-30T06:35:57.343ZDEBGtest_metrics (BackgroundTasks): roles background_task = bfd_manager roles = RoleSet { roles: {} }
110552025-04-30T06:35:57.344ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = bfd_manager resource = Database result = Ok(())
110562025-04-30T06:35:57.354ZDEBGtest_metrics (BackgroundTasks): activating background_task = phantom_disks iteration = 2 reason = Timeout
110572025-04-30T06:35:57.354ZDEBGtest_metrics (BackgroundTasks): activating background_task = physical_disk_adoption iteration = 3 reason = Timeout
110582025-04-30T06:35:57.354ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = physical_disk_adoption elapsed = 33.266\u{b5}s iteration = 3 status = {"error":"task disabled"}
110592025-04-30T06:35:57.354ZDEBGtest_metrics (BackgroundTasks): activating background_task = service_zone_nat_tracker iteration = 2 reason = Timeout
110602025-04-30T06:35:57.354ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory
110612025-04-30T06:35:57.354ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110622025-04-30T06:35:57.354ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
110632025-04-30T06:35:57.354ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
110642025-04-30T06:35:57.355ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
110652025-04-30T06:35:57.355ZDEBGtest_metrics (BackgroundTasks): activating background_task = switch_port_config_manager iteration = 2 reason = Timeout
110662025-04-30T06:35:57.355ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet
110672025-04-30T06:35:57.355ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = switch_port_config_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110682025-04-30T06:35:57.355ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
110692025-04-30T06:35:57.355ZDEBGtest_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
110702025-04-30T06:35:57.356ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
110712025-04-30T06:35:57.356ZDEBGtest_metrics (BackgroundTasks): activating background_task = v2p_manager iteration = 2 reason = Timeout
110722025-04-30T06:35:57.356ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
110732025-04-30T06:35:57.356ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
110742025-04-30T06:35:57.356ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
110752025-04-30T06:35:57.357ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110762025-04-30T06:35:57.358ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4fbcf1e8-f928-43e5-a66c-5106153ac64f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110772025-04-30T06:35:57.358ZDEBGtest_metrics (BackgroundTasks): activating background_task = instance_watcher iteration = 2 reason = Timeout
110782025-04-30T06:35:57.358ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet
110792025-04-30T06:35:57.358ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = instance_watcher resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110802025-04-30T06:35:57.358ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
110812025-04-30T06:35:57.358ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
110822025-04-30T06:35:57.359ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
110832025-04-30T06:35:57.359ZDEBGtest_metrics (BackgroundTasks): activating background_task = instance_updater iteration = 2 reason = Timeout
110842025-04-30T06:35:57.359ZINFOtest_metrics (BackgroundTasks): background instance updater explicitly disabled background_task = instance_updater
110852025-04-30T06:35:57.359ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = instance_updater elapsed = 65.53\u{b5}s iteration = 2 status = {"destroyed_active_vmms":0,"disabled":true,"failed_active_vmms":0,"query_errors":[],"saga_errors":[],"sagas_completed":0,"sagas_started":0,"terminated_active_migrations":0}
110862025-04-30T06:35:57.359ZDEBGtest_metrics (BackgroundTasks): activating background_task = vpc_route_manager iteration = 2 reason = Timeout
110872025-04-30T06:35:57.359ZINFOtest_metrics (BackgroundTasks): VPC route manager running background_task = vpc_route_manager
110882025-04-30T06:35:57.359ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
110892025-04-30T06:35:57.359ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
110902025-04-30T06:35:57.359ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
110912025-04-30T06:35:57.359ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
110922025-04-30T06:35:57.360ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
110932025-04-30T06:35:57.360ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4fbcf1e8-f928-43e5-a66c-5106153ac64f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110942025-04-30T06:35:57.360ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4fbcf1e8-f928-43e5-a66c-5106153ac64f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110952025-04-30T06:35:57.361ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4fbcf1e8-f928-43e5-a66c-5106153ac64f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
110962025-04-30T06:35:57.361ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3072cdab-5624-41ae-b686-85e37489b99f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
110972025-04-30T06:35:57.361ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
110982025-04-30T06:35:57.362ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
110992025-04-30T06:35:57.362ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
111002025-04-30T06:35:57.362ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
111012025-04-30T06:35:57.364ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
111022025-04-30T06:35:57.368ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 04 9c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:50108 req_id = 040f309b-2311-4c12-9657-9713f7f26090
111032025-04-30T06:35:57.368ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1180,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:50108 req_id = 040f309b-2311-4c12-9657-9713f7f26090
111042025-04-30T06:35:57.368ZDEBGtest_metrics (store): query key key = _dendrite._tcp
111052025-04-30T06:35:57.368ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111062025-04-30T06:35:57.369ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:50108 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 040f309b-2311-4c12-9657-9713f7f26090
111072025-04-30T06:35:57.369ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
111082025-04-30T06:35:57.370ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = phantom_disks elapsed = 16.639844ms iteration = 2 status = {"phantom_disk_deleted_err":0,"phantom_disk_deleted_ok":0}
111092025-04-30T06:35:57.381ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3822, tv_nsec: 180116732 } })
111102025-04-30T06:35:57.381ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
111112025-04-30T06:35:57.383ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111122025-04-30T06:35:57.390ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Inventory result = Ok(())
111132025-04-30T06:35:57.390ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
111142025-04-30T06:35:57.390ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
111152025-04-30T06:35:57.390ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
111162025-04-30T06:35:57.390ZDEBGtest_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111172025-04-30T06:35:57.394ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Fleet result = Ok(())
111182025-04-30T06:35:57.394ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database
111192025-04-30T06:35:57.394ZDEBGtest_metrics (BackgroundTasks): roles background_task = switch_port_config_manager roles = RoleSet { roles: {} }
111202025-04-30T06:35:57.394ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = switch_port_config_manager resource = Database result = Ok(())
111212025-04-30T06:35:57.398ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: df fb 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49174 req_id = e4dfc9f8-56bd-4fe1-be59-a199deb23019
111222025-04-30T06:35:57.398ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57339,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49174 req_id = e4dfc9f8-56bd-4fe1-be59-a199deb23019
111232025-04-30T06:35:57.398ZDEBGtest_metrics (store): query key key = _dendrite._tcp
111242025-04-30T06:35:57.398ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111252025-04-30T06:35:57.399ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49174 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e4dfc9f8-56bd-4fe1-be59-a199deb23019
111262025-04-30T06:35:57.405ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111272025-04-30T06:35:57.451ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Fleet result = Ok(())
111282025-04-30T06:35:57.461ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database
111292025-04-30T06:35:57.461ZDEBGtest_metrics (BackgroundTasks): roles background_task = instance_watcher roles = RoleSet { roles: {} }
111302025-04-30T06:35:57.462ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = instance_watcher resource = Database result = Ok(())
111312025-04-30T06:35:57.480ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111322025-04-30T06:35:57.480ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111332025-04-30T06:35:57.481ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111342025-04-30T06:35:57.489ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3822, tv_nsec: 288143539 } })
111352025-04-30T06:35:57.489ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
111362025-04-30T06:35:57.492ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 5a 90 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e Z...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63060 req_id = 25b8b1e3-8e64-45f4-8187-dca61a976a22
111372025-04-30T06:35:57.492ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 23184,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63060 req_id = 25b8b1e3-8e64-45f4-8187-dca61a976a22
111382025-04-30T06:35:57.492ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111392025-04-30T06:35:57.493ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63060 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 25b8b1e3-8e64-45f4-8187-dca61a976a22
111402025-04-30T06:35:57.494ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111412025-04-30T06:35:57.498ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
111422025-04-30T06:35:57.499ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet
111432025-04-30T06:35:57.499ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = vpc_route_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111442025-04-30T06:35:57.499ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
111452025-04-30T06:35:57.500ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
111462025-04-30T06:35:57.501ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
111472025-04-30T06:35:57.502ZINFOtest_metrics (dropshot_internal): request completed latency_us = 144371 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 4fbcf1e8-f928-43e5-a66c-5106153ac64f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
111482025-04-30T06:35:57.504ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
111492025-04-30T06:35:57.589ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
111502025-04-30T06:35:57.591ZDEBGtest_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
111512025-04-30T06:35:57.591ZDEBGtest_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
111522025-04-30T06:35:57.591ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
111532025-04-30T06:35:57.593ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
111542025-04-30T06:35:57.594ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "4fbcf1e8-f928-43e5-a66c-5106153ac64f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
111552025-04-30T06:35:57.594ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111562025-04-30T06:35:57.595ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
111572025-04-30T06:35:57.595ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
111582025-04-30T06:35:57.596ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
111592025-04-30T06:35:57.598ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a36209b5-92a8-4bd0-a040-e27b13c3d41f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111602025-04-30T06:35:57.598ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111612025-04-30T06:35:57.602ZTRACtest_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111622025-04-30T06:35:57.602ZTRACtest_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
111632025-04-30T06:35:57.603ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 80 86 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62288 req_id = 2c7e3d79-3255-4462-8029-b531ca35a88e
111642025-04-30T06:35:57.604ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32902,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62288 req_id = 2c7e3d79-3255-4462-8029-b531ca35a88e
111652025-04-30T06:35:57.604ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111662025-04-30T06:35:57.606ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62288 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 2c7e3d79-3255-4462-8029-b531ca35a88e
111672025-04-30T06:35:57.608ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a36209b5-92a8-4bd0-a040-e27b13c3d41f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111682025-04-30T06:35:57.612ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a36209b5-92a8-4bd0-a040-e27b13c3d41f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111692025-04-30T06:35:57.613ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a36209b5-92a8-4bd0-a040-e27b13c3d41f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
111702025-04-30T06:35:57.613ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111712025-04-30T06:35:57.614ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
111722025-04-30T06:35:57.620ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
111732025-04-30T06:35:57.688ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
111742025-04-30T06:35:57.688ZDEBGtest_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
111752025-04-30T06:35:57.688ZDEBGtest_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
111762025-04-30T06:35:57.688ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
111772025-04-30T06:35:57.690ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
111782025-04-30T06:35:57.691ZINFOtest_metrics (BackgroundTasks): all instance checks complete background_task = instance_watcher pruned_instances = 0 total_completed = 0 total_failed = 0 total_incomplete = 0 total_instances = 0 update_sagas_queued = 0
111792025-04-30T06:35:57.691ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = instance_watcher elapsed = 333.308134ms iteration = 2 status = {"failed_checks":{},"incomplete_checks":{},"instance_states":{},"pruned_instances":0,"total_instances":0,"update_sagas_queued":0}
111802025-04-30T06:35:57.696ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 27 6c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 'l..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62809 req_id = 3b3e246a-9347-42ad-971c-b05b3ffb812e
111812025-04-30T06:35:57.696ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10092,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62809 req_id = 3b3e246a-9347-42ad-971c-b05b3ffb812e
111822025-04-30T06:35:57.697ZDEBGtest_metrics (store): query key key = _dendrite._tcp
111832025-04-30T06:35:57.697ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
111842025-04-30T06:35:57.697ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62809 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3b3e246a-9347-42ad-971c-b05b3ffb812e
111852025-04-30T06:35:57.697ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111862025-04-30T06:35:57.698ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111872025-04-30T06:35:57.698ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111882025-04-30T06:35:57.698ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
111892025-04-30T06:35:57.698ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
111902025-04-30T06:35:57.703ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Fleet result = Ok(())
111912025-04-30T06:35:57.704ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
111922025-04-30T06:35:57.704ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
111932025-04-30T06:35:57.706ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
111942025-04-30T06:35:57.708ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
111952025-04-30T06:35:57.708ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
111962025-04-30T06:35:57.708ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
111972025-04-30T06:35:57.708ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
111982025-04-30T06:35:57.708ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
111992025-04-30T06:35:57.710ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3822, tv_nsec: 509271001 } })
112002025-04-30T06:35:57.710ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
112012025-04-30T06:35:57.711ZINFOtest_metrics (dropshot_internal): request completed latency_us = 113792 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = a36209b5-92a8-4bd0-a040-e27b13c3d41f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
112022025-04-30T06:35:57.766ZINFOtest_metrics (BackgroundTasks): VPC route manager sled b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager
112032025-04-30T06:35:57.766ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = None method = GET uri = http://[::1]:56237/vpc-routes
112042025-04-30T06:35:57.767ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112052025-04-30T06:35:57.770ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
112062025-04-30T06:35:57.770ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet
112072025-04-30T06:35:57.771ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = v2p_manager resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112082025-04-30T06:35:57.771ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
112092025-04-30T06:35:57.771ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112102025-04-30T06:35:57.771ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
112112025-04-30T06:35:57.773ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:45834 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112122025-04-30T06:35:57.775ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "a36209b5-92a8-4bd0-a040-e27b13c3d41f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
112132025-04-30T06:35:57.775ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
112142025-04-30T06:35:57.775ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
112152025-04-30T06:35:57.776ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
112162025-04-30T06:35:57.778ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:45834 req_id = 8e9ec892-8aef-4ad6-928a-ebd52d153a39 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
112172025-04-30T06:35:57.778ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a2 87 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60623 req_id = de7b8d90-1fe9-496a-8bd1-f1de13dfdad2
112182025-04-30T06:35:57.778ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 41607,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60623 req_id = de7b8d90-1fe9-496a-8bd1-f1de13dfdad2
112192025-04-30T06:35:57.779ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
112202025-04-30T06:35:57.779ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60623 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = de7b8d90-1fe9-496a-8bd1-f1de13dfdad2
112212025-04-30T06:35:57.779ZINFOtest_metrics (dropshot): request completed latency_us = 838 local_addr = [::1]:56237 method = GET remote_addr = [::1]:45834 req_id = 8e9ec892-8aef-4ad6-928a-ebd52d153a39 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /vpc-routes
112222025-04-30T06:35:57.779ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112232025-04-30T06:35:57.786ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Fleet result = Ok(())
112242025-04-30T06:35:57.786ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database
112252025-04-30T06:35:57.786ZDEBGtest_metrics (BackgroundTasks): roles background_task = v2p_manager roles = RoleSet { roles: {} }
112262025-04-30T06:35:57.787ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = v2p_manager resource = Database result = Ok(())
112272025-04-30T06:35:57.789ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
112282025-04-30T06:35:57.842ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
112292025-04-30T06:35:57.842ZDEBGtest_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
112302025-04-30T06:35:57.843ZDEBGtest_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
112312025-04-30T06:35:57.843ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
112322025-04-30T06:35:57.844ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
112332025-04-30T06:35:57.846ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:56237/vpc-routes", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e9ec892-8aef-4ad6-928a-ebd52d153a39", "content-length": "2", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
112342025-04-30T06:35:57.846ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database
112352025-04-30T06:35:57.846ZDEBGtest_metrics (BackgroundTasks): roles background_task = vpc_route_manager roles = RoleSet { roles: {} }
112362025-04-30T06:35:57.847ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = vpc_route_manager resource = Database result = Ok(())
112372025-04-30T06:35:57.859ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba658ab6-71aa-4f72-89ef-c09e44a9843d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
112382025-04-30T06:35:57.878ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
112392025-04-30T06:35:57.887ZINFOtest_metrics (BackgroundTasks): computed internet gateway mappings for sled assocs = {} background_task = vpc_route_manager sled = sim-b6d65341
112402025-04-30T06:35:57.887ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager body = Some(Body) method = PUT uri = http://[::1]:56237/eip-gateways
112412025-04-30T06:35:57.901ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = PUT remote_addr = [::1]:45834 req_id = bde5293c-d706-480a-9fda-57e8088289c8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
112422025-04-30T06:35:57.901ZINFOtest_metrics (dropshot): request completed latency_us = 501 local_addr = [::1]:56237 method = PUT remote_addr = [::1]:45834 req_id = bde5293c-d706-480a-9fda-57e8088289c8 response_code = 204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /eip-gateways
112432025-04-30T06:35:57.905ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = vpc_route_manager result = Ok(Response { url: "http://[::1]:56237/eip-gateways", status: 204, headers: {"x-request-id": "bde5293c-d706-480a-9fda-57e8088289c8", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
112442025-04-30T06:35:57.913ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = vpc_route_manager elapsed = 553.627606ms iteration = 2 status = {}
112452025-04-30T06:35:57.933ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
112462025-04-30T06:35:57.937ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d06f0be-5edb-4eed-9356-40c5de5796cd", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
112472025-04-30T06:35:57.938ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
112482025-04-30T06:35:57.942ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "abb86c0e-166f-451a-b366-2aba6ce74e8d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:57 GMT"} })
112492025-04-30T06:35:57.942ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
112502025-04-30T06:35:58.040ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager body = None method = GET uri = http://[::1]:56237/v2p
112512025-04-30T06:35:58.041ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
112522025-04-30T06:35:58.041ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
112532025-04-30T06:35:58.042ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
112542025-04-30T06:35:58.043ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
112552025-04-30T06:35:58.043ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112562025-04-30T06:35:58.043ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112572025-04-30T06:35:58.043ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112582025-04-30T06:35:58.045ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112592025-04-30T06:35:58.045ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:40613 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
112602025-04-30T06:35:58.046ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:40613 req_id = 7e605dd9-c742-4f64-991e-15e9ae0c1af7 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
112612025-04-30T06:35:58.047ZINFOtest_metrics (dropshot): request completed latency_us = 930 local_addr = [::1]:56237 method = GET remote_addr = [::1]:40613 req_id = 7e605dd9-c742-4f64-991e-15e9ae0c1af7 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /v2p
112622025-04-30T06:35:58.047ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = b6d65341-167c-41df-9b5c-41cded99c229 background_task = v2p_manager result = Ok(Response { url: "http://[::1]:56237/v2p", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e605dd9-c742-4f64-991e-15e9ae0c1af7", "content-length": "2", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112632025-04-30T06:35:58.047ZINFOtest_metrics (BackgroundTasks): found opte v2p mappings background_task = v2p_manager interfaces = {} sled = sim-b6d65341
112642025-04-30T06:35:58.047ZINFOtest_metrics (BackgroundTasks): v2p mappings to delete background_task = v2p_manager mappings = [] sled = sim-b6d65341
112652025-04-30T06:35:58.047ZINFOtest_metrics (BackgroundTasks): v2p mappings to add background_task = v2p_manager mappings = [] sled = sim-b6d65341
112662025-04-30T06:35:58.049ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = v2p_manager elapsed = 693.219425ms iteration = 2 status = {}
112672025-04-30T06:35:58.049ZTRACtest_metrics (BackgroundTasks): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) }
112682025-04-30T06:35:58.049ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = service_zone_nat_tracker resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
112692025-04-30T06:35:58.049ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database
112702025-04-30T06:35:58.049ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {} }
112712025-04-30T06:35:58.050ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Database result = Ok(())
112722025-04-30T06:35:58.053ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112732025-04-30T06:35:58.058ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
112742025-04-30T06:35:58.058ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112752025-04-30T06:35:58.058ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112762025-04-30T06:35:58.059ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112772025-04-30T06:35:58.059ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "04881672-44ff-4ba9-ba38-d9647855d7e9", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112782025-04-30T06:35:58.059ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
112792025-04-30T06:35:58.059ZDEBGtest_metrics (BackgroundTasks): roles background_task = service_zone_nat_tracker roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
112802025-04-30T06:35:58.063ZDEBGtest_metrics (BackgroundTasks): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = service_zone_nat_tracker resource = Sled { parent: Fleet, key: b6d65341-167c-41df-9b5c-41cded99c229, lookup_type: ById(b6d65341-167c-41df-9b5c-41cded99c229) } result = Ok(())
112812025-04-30T06:35:58.064ZERROtest_metrics (BackgroundTasks): ipv6 addresses for service zone nat not implemented background_task = service_zone_nat_tracker
112822025-04-30T06:35:58.064ZERROtest_metrics (BackgroundTasks): generated config for fewer than the minimum allowed number of dns zones background_task = service_zone_nat_tracker
112832025-04-30T06:35:58.064ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = service_zone_nat_tracker elapsed = 709.662796ms iteration = 2 status = {"error":"generated config for fewer than the minimum allowed number of dns zones"}
112842025-04-30T06:35:58.065ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
112852025-04-30T06:35:58.065ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
112862025-04-30T06:35:58.066ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
112872025-04-30T06:35:58.067ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fec8edd5-9f64-4060-a508-a0e69fcb235a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112882025-04-30T06:35:58.068ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
112892025-04-30T06:35:58.069ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8f8eb49d-3563-4cfc-a696-73ad8762b8b0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112902025-04-30T06:35:58.069ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
112912025-04-30T06:35:58.070ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4afd0823-40c8-4948-9a64-98d0858870f8", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112922025-04-30T06:35:58.070ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
112932025-04-30T06:35:58.071ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a89a58c-a7e0-4eed-9732-ae9e26d8ac00", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112942025-04-30T06:35:58.071ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
112952025-04-30T06:35:58.075ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "578d53a0-357a-48e4-bdbd-9cbbbd6b500f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112962025-04-30T06:35:58.075ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
112972025-04-30T06:35:58.079ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ef2da7a3-60c9-41bf-b716-c9676a312b99", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
112982025-04-30T06:35:58.079ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
112992025-04-30T06:35:58.083ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "017e4b5d-519e-4424-b839-4d4f230cd232", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113002025-04-30T06:35:58.083ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
113012025-04-30T06:35:58.093ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b778ceb6-53cb-41ce-8121-0de9fd7d5f87", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113022025-04-30T06:35:58.093ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
113032025-04-30T06:35:58.126ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "206b4cb6-556b-4422-8ec3-a5bae2418ad9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113042025-04-30T06:35:58.127ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
113052025-04-30T06:35:58.127ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
113062025-04-30T06:35:58.134ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2294e31d-3105-4ec8-9a2f-94fc80bca48f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113072025-04-30T06:35:58.135ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
113082025-04-30T06:35:58.139ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "03350097-cb3f-4863-92ce-c71eaf26c76c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113092025-04-30T06:35:58.139ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
113102025-04-30T06:35:58.141ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa4bb453-08a9-4247-bfea-5a6f2b8c1230", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113112025-04-30T06:35:58.142ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
113122025-04-30T06:35:58.143ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a84c9f03-557b-4994-9ec3-4eb920542693", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113132025-04-30T06:35:58.143ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
113142025-04-30T06:35:58.144ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d0443a3d-28b9-4000-965e-094cfc19d27e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113152025-04-30T06:35:58.144ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
113162025-04-30T06:35:58.146ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26ed3ba0-e2cc-48d3-ada4-74927c7fb2af", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113172025-04-30T06:35:58.146ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
113182025-04-30T06:35:58.147ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ced1e58d-6240-4c3e-ad06-f9d280e00811", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113192025-04-30T06:35:58.147ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113202025-04-30T06:35:58.147ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113212025-04-30T06:35:58.147ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113222025-04-30T06:35:58.147ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113232025-04-30T06:35:58.147ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113242025-04-30T06:35:58.158ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113252025-04-30T06:35:58.168ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113262025-04-30T06:35:58.168ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
113272025-04-30T06:35:58.168ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113282025-04-30T06:35:58.168ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113292025-04-30T06:35:58.168ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113302025-04-30T06:35:58.169ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113312025-04-30T06:35:58.172ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113322025-04-30T06:35:58.175ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
113332025-04-30T06:35:58.175ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113342025-04-30T06:35:58.175ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113352025-04-30T06:35:58.176ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113362025-04-30T06:35:58.178ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113372025-04-30T06:35:58.178ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113382025-04-30T06:35:58.178ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113392025-04-30T06:35:58.179ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113402025-04-30T06:35:58.179ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113412025-04-30T06:35:58.235ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
113422025-04-30T06:35:58.235ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
113432025-04-30T06:35:58.236ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:41822 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
113442025-04-30T06:35:58.237ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:41822 req_id = 36dc814d-3091-44d9-b707-3317af7345f4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113452025-04-30T06:35:58.238ZINFOtest_metrics (dropshot): request completed latency_us = 956 local_addr = [::1]:56237 method = GET remote_addr = [::1]:41822 req_id = 36dc814d-3091-44d9-b707-3317af7345f4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
113462025-04-30T06:35:58.238ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113472025-04-30T06:35:58.245ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
113482025-04-30T06:35:58.245ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113492025-04-30T06:35:58.246ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113502025-04-30T06:35:58.246ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113512025-04-30T06:35:58.247ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "36dc814d-3091-44d9-b707-3317af7345f4", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
113522025-04-30T06:35:58.249ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113532025-04-30T06:35:58.249ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
113542025-04-30T06:35:58.249ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
113552025-04-30T06:35:58.253ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113562025-04-30T06:35:58.253ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113572025-04-30T06:35:58.253ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113582025-04-30T06:35:58.253ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113592025-04-30T06:35:58.254ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113602025-04-30T06:35:58.258ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
113612025-04-30T06:35:58.258ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
113622025-04-30T06:35:58.258ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
113632025-04-30T06:35:58.259ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113642025-04-30T06:35:58.265ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113652025-04-30T06:35:58.265ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113662025-04-30T06:35:58.265ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113672025-04-30T06:35:58.266ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113682025-04-30T06:35:58.334ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
113692025-04-30T06:35:58.338ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
113702025-04-30T06:35:58.339ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
113712025-04-30T06:35:58.339ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.444604821s iteration = 39 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
113722025-04-30T06:35:58.339ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 40 reason = Signaled
113732025-04-30T06:35:58.339ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
113742025-04-30T06:35:58.339ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113752025-04-30T06:35:58.339ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113762025-04-30T06:35:58.339ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113772025-04-30T06:35:58.340ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113782025-04-30T06:35:58.345ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
113792025-04-30T06:35:58.349ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
113802025-04-30T06:35:58.349ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
113812025-04-30T06:35:58.349ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
113822025-04-30T06:35:58.350ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
113832025-04-30T06:35:58.361ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
113842025-04-30T06:35:58.361ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
113852025-04-30T06:35:58.366ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 07 63 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .c..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52989 req_id = 96b34f69-cdc3-4e91-894b-29648ca13b77
113862025-04-30T06:35:58.366ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 1891,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52989 req_id = 96b34f69-cdc3-4e91-894b-29648ca13b77
113872025-04-30T06:35:58.366ZDEBGtest_metrics (store): query key key = _mgs._tcp
113882025-04-30T06:35:58.367ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113892025-04-30T06:35:58.367ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52989 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 96b34f69-cdc3-4e91-894b-29648ca13b77
113902025-04-30T06:35:58.367ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 166836888 } })
113912025-04-30T06:35:58.367ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
113922025-04-30T06:35:58.372ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 41 44 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e AD..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55853 req_id = 9126217a-e9ff-4c82-a080-258aa591fbd1
113932025-04-30T06:35:58.372ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 16708,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55853 req_id = 9126217a-e9ff-4c82-a080-258aa591fbd1
113942025-04-30T06:35:58.372ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
113952025-04-30T06:35:58.372ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55853 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 9126217a-e9ff-4c82-a080-258aa591fbd1
113962025-04-30T06:35:58.478ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
113972025-04-30T06:35:58.478ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
113982025-04-30T06:35:58.478ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
113992025-04-30T06:35:58.479ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114002025-04-30T06:35:58.479ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114012025-04-30T06:35:58.479ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114022025-04-30T06:35:58.480ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 36 0a 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c 6............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58492 req_id = e00b3d77-974a-4d1b-9c1a-a5617b72f1bc
114032025-04-30T06:35:58.480ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13834,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58492 req_id = e00b3d77-974a-4d1b-9c1a-a5617b72f1bc
114042025-04-30T06:35:58.480ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
114052025-04-30T06:35:58.480ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 13834, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58492 req_id = e00b3d77-974a-4d1b-9c1a-a5617b72f1bc
114062025-04-30T06:35:58.480ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
114072025-04-30T06:35:58.481ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
114082025-04-30T06:35:58.481ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
114092025-04-30T06:35:58.483ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
114102025-04-30T06:35:58.488ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
114112025-04-30T06:35:58.488ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114122025-04-30T06:35:58.488ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114132025-04-30T06:35:58.489ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114142025-04-30T06:35:58.489ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "8458e03d-61f6-4489-8675-22082e84d37d", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114152025-04-30T06:35:58.489ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
114162025-04-30T06:35:58.492ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
114172025-04-30T06:35:58.492ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
114182025-04-30T06:35:58.493ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
114192025-04-30T06:35:58.493ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee806c36-751d-4986-beef-6ea3e1ad5322", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114202025-04-30T06:35:58.493ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
114212025-04-30T06:35:58.496ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9a4d507-5a57-407f-978b-5deffa310f8d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114222025-04-30T06:35:58.497ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
114232025-04-30T06:35:58.506ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0fe7c4dc-86b6-434c-bcc8-b8ceb38589ed", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114242025-04-30T06:35:58.506ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
114252025-04-30T06:35:58.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "068d2c83-fc6c-4343-ac31-9bb4c1da4b69", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114262025-04-30T06:35:58.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
114272025-04-30T06:35:58.511ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5f79e0e-5d63-4b1c-867a-c183184f0e91", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114282025-04-30T06:35:58.511ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
114292025-04-30T06:35:58.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114302025-04-30T06:35:58.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114312025-04-30T06:35:58.514ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114322025-04-30T06:35:58.514ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114332025-04-30T06:35:58.514ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114342025-04-30T06:35:58.514ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
114352025-04-30T06:35:58.516ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb5df177-59d4-4701-b656-990dd57bff7c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114362025-04-30T06:35:58.516ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
114372025-04-30T06:35:58.519ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "21892e90-08b5-4402-b66f-caff994c9d3e", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114382025-04-30T06:35:58.520ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
114392025-04-30T06:35:58.521ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6d3bc113-df55-4c08-a1ed-34b0f430a24f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114402025-04-30T06:35:58.521ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
114412025-04-30T06:35:58.524ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "14ae6f78-1359-419f-ade9-6b42c25aeaa0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114422025-04-30T06:35:58.524ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
114432025-04-30T06:35:58.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0caaf1ea-1206-41de-a8a3-c562a7235d81", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114442025-04-30T06:35:58.530ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
114452025-04-30T06:35:58.557ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
114462025-04-30T06:35:58.558ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a935dcd7-3867-43d6-9db8-158dbafe595b", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114472025-04-30T06:35:58.558ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
114482025-04-30T06:35:58.560ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "285d36d4-e6dc-4eb8-8cd1-67e57630d333", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114492025-04-30T06:35:58.561ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
114502025-04-30T06:35:58.563ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4065cc15-517e-48ff-931f-e37771aa8ecb", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114512025-04-30T06:35:58.563ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
114522025-04-30T06:35:58.565ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3481fb1e-04a1-4507-b651-0b7820bd84d1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114532025-04-30T06:35:58.565ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
114542025-04-30T06:35:58.567ZDEBGtest_metrics (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114552025-04-30T06:35:58.567ZDEBGtest_metrics (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114562025-04-30T06:35:58.567ZDEBGtest_metrics (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
114572025-04-30T06:35:58.568ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
114582025-04-30T06:35:58.569ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c6ec3a6-b8cc-4d03-802a-3912aafce208", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114592025-04-30T06:35:58.569ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
114602025-04-30T06:35:58.573ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
114612025-04-30T06:35:58.573ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
114622025-04-30T06:35:58.575ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cdab1812-36c0-4438-9915-fb963cc01e05", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114632025-04-30T06:35:58.575ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
114642025-04-30T06:35:58.576ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = measurements_i64
114652025-04-30T06:35:58.576ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
114662025-04-30T06:35:58.578ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e07bb9f-3f7a-4e77-a0e7-0e3ecfa4cb0b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114672025-04-30T06:35:58.578ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
114682025-04-30T06:35:58.580ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "acc57a8e-a2a9-4e5b-8090-0215f3f2840d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114692025-04-30T06:35:58.580ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
114702025-04-30T06:35:58.581ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "12d05921-b8a4-4198-9edb-1df6a5cb0ebe", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114712025-04-30T06:35:58.581ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
114722025-04-30T06:35:58.582ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e29aaf69-d802-4783-bdb7-370179dbf772", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114732025-04-30T06:35:58.582ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
114742025-04-30T06:35:58.583ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c724618d-d961-4736-b978-5001ed2faf78", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114752025-04-30T06:35:58.584ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
114762025-04-30T06:35:58.585ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0f499b28-c560-4e99-aa61-2cae2b5eb848", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114772025-04-30T06:35:58.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
114782025-04-30T06:35:58.587ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "403b524c-29bd-48e2-8a3f-7cf76e9c26f8", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114792025-04-30T06:35:58.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
114802025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "79119bd2-0bd9-4072-9db7-4461b11403ff", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114812025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
114822025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "122491c3-22fe-4c50-a115-eeee12096dac", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114832025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
114842025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6cec9978-4738-4800-9d9a-7c5ae87e929b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114852025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
114862025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "87c9efca-8394-4cfc-a226-3aa7d56ed35b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114872025-04-30T06:35:58.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
114882025-04-30T06:35:58.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "43ac29d9-5f7b-4e59-88a5-6757aa08a017", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114892025-04-30T06:35:58.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
114902025-04-30T06:35:58.598ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee93d1e6-3024-4604-b3e3-946e9765be37", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114912025-04-30T06:35:58.598ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
114922025-04-30T06:35:58.599ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "807a4055-d5e5-4910-a5bc-279ff5750620", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114932025-04-30T06:35:58.599ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
114942025-04-30T06:35:58.600ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "352f8de6-aea2-4d72-b32e-cff185598dc0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114952025-04-30T06:35:58.600ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
114962025-04-30T06:35:58.601ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92ce54af-39e4-450e-a284-6937d5f78637", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114972025-04-30T06:35:58.601ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
114982025-04-30T06:35:58.602ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "383cdbc1-21a7-48fb-94ca-41af8a79ca99", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
114992025-04-30T06:35:58.603ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
115002025-04-30T06:35:58.604ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1f5b9ec-5907-4e50-9273-1268d2cfc348", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115012025-04-30T06:35:58.604ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
115022025-04-30T06:35:58.606ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "363b235a-cd46-48eb-81f1-aeaab0b80240", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115032025-04-30T06:35:58.607ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
115042025-04-30T06:35:58.608ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115052025-04-30T06:35:58.608ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115062025-04-30T06:35:58.608ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115072025-04-30T06:35:58.608ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115082025-04-30T06:35:58.609ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115092025-04-30T06:35:58.611ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b11e02cd-521d-4a6d-83e3-627b28860bea", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115102025-04-30T06:35:58.611ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
115112025-04-30T06:35:58.612ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115122025-04-30T06:35:58.618ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115132025-04-30T06:35:58.618ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115142025-04-30T06:35:58.618ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115152025-04-30T06:35:58.619ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115162025-04-30T06:35:58.622ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115172025-04-30T06:35:58.622ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115182025-04-30T06:35:58.623ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115192025-04-30T06:35:58.623ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59c81247-0af7-4053-81ae-cbc15f0e5768", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115202025-04-30T06:35:58.623ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
115212025-04-30T06:35:58.627ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1ed15fbd-9e89-49a6-80a4-d5aba85e5d9a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115222025-04-30T06:35:58.627ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
115232025-04-30T06:35:58.632ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cce72f65-7fb5-44e6-b7a8-47f906f0678a", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115242025-04-30T06:35:58.632ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
115252025-04-30T06:35:58.636ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "64e5903e-d0fb-4940-9125-2dabf4f7138c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115262025-04-30T06:35:58.636ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
115272025-04-30T06:35:58.638ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b83742d-cf83-4020-9618-c26901efb18d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115282025-04-30T06:35:58.638ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
115292025-04-30T06:35:58.639ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "167e3f55-5ec6-4451-b301-2bf4ca475a24", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115302025-04-30T06:35:58.640ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
115312025-04-30T06:35:58.642ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ecd53a9b-8045-4a79-a0aa-30170eae0d1a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115322025-04-30T06:35:58.642ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
115332025-04-30T06:35:58.644ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3be6abf-1b49-445b-89bb-e25f6312c9d2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115342025-04-30T06:35:58.644ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115352025-04-30T06:35:58.644ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115362025-04-30T06:35:58.644ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115372025-04-30T06:35:58.644ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115382025-04-30T06:35:58.644ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115392025-04-30T06:35:58.649ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115402025-04-30T06:35:58.654ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115412025-04-30T06:35:58.654ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
115422025-04-30T06:35:58.654ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115432025-04-30T06:35:58.654ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115442025-04-30T06:35:58.654ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115452025-04-30T06:35:58.655ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115462025-04-30T06:35:58.661ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115472025-04-30T06:35:58.665ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
115482025-04-30T06:35:58.665ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115492025-04-30T06:35:58.665ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115502025-04-30T06:35:58.665ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115512025-04-30T06:35:58.666ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
115522025-04-30T06:35:58.668ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
115532025-04-30T06:35:58.717ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
115542025-04-30T06:35:58.717ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
115552025-04-30T06:35:58.718ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
115562025-04-30T06:35:58.718ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115572025-04-30T06:35:58.718ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115582025-04-30T06:35:58.718ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115592025-04-30T06:35:58.719ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115602025-04-30T06:35:58.719ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:35374 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
115612025-04-30T06:35:58.720ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:35374 req_id = b04c8948-7f7a-4771-9a35-4071c61a15b8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115622025-04-30T06:35:58.720ZINFOtest_metrics (dropshot): request completed latency_us = 502 local_addr = [::1]:56237 method = GET remote_addr = [::1]:35374 req_id = b04c8948-7f7a-4771-9a35-4071c61a15b8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
115632025-04-30T06:35:58.721ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "b04c8948-7f7a-4771-9a35-4071c61a15b8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
115642025-04-30T06:35:58.725ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115652025-04-30T06:35:58.725ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
115662025-04-30T06:35:58.725ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
115672025-04-30T06:35:58.727ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115682025-04-30T06:35:58.727ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115692025-04-30T06:35:58.727ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115702025-04-30T06:35:58.727ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115712025-04-30T06:35:58.727ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115722025-04-30T06:35:58.730ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115732025-04-30T06:35:58.734ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
115742025-04-30T06:35:58.734ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115752025-04-30T06:35:58.734ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115762025-04-30T06:35:58.734ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115772025-04-30T06:35:58.737ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115782025-04-30T06:35:58.740ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115792025-04-30T06:35:58.741ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115802025-04-30T06:35:58.741ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115812025-04-30T06:35:58.742ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115822025-04-30T06:35:58.744ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
115832025-04-30T06:35:58.744ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
115842025-04-30T06:35:58.747ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
115852025-04-30T06:35:58.782ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
115862025-04-30T06:35:58.783ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
115872025-04-30T06:35:58.786ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 447.180889ms iteration = 40 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
115882025-04-30T06:35:58.786ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 41 reason = Signaled
115892025-04-30T06:35:58.787ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
115902025-04-30T06:35:58.787ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
115912025-04-30T06:35:58.787ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115922025-04-30T06:35:58.787ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115932025-04-30T06:35:58.788ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
115942025-04-30T06:35:58.794ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
115952025-04-30T06:35:58.794ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
115962025-04-30T06:35:58.800ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
115972025-04-30T06:35:58.800ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
115982025-04-30T06:35:58.800ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
115992025-04-30T06:35:58.800ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
116002025-04-30T06:35:58.805ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
116012025-04-30T06:35:58.805ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
116022025-04-30T06:35:58.806ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 44 4d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 DM..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:53469 req_id = c2403493-f4a7-4bc2-9e44-0d3cb18f1277
116032025-04-30T06:35:58.806ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17485,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53469 req_id = c2403493-f4a7-4bc2-9e44-0d3cb18f1277
116042025-04-30T06:35:58.806ZDEBGtest_metrics (store): query key key = _mgs._tcp
116052025-04-30T06:35:58.806ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116062025-04-30T06:35:58.806ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53469 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = c2403493-f4a7-4bc2-9e44-0d3cb18f1277
116072025-04-30T06:35:58.807ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 606102877 } })
116082025-04-30T06:35:58.807ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
116092025-04-30T06:35:58.807ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1d 18 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:56418 req_id = af9fa303-ea1f-4a75-b3e0-489aeac0eaa6
116102025-04-30T06:35:58.807ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7448,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56418 req_id = af9fa303-ea1f-4a75-b3e0-489aeac0eaa6
116112025-04-30T06:35:58.807ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
116122025-04-30T06:35:58.807ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:56418 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = af9fa303-ea1f-4a75-b3e0-489aeac0eaa6
116132025-04-30T06:35:58.851ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
116142025-04-30T06:35:58.852ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116152025-04-30T06:35:58.852ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116162025-04-30T06:35:58.852ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116172025-04-30T06:35:58.852ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116182025-04-30T06:35:58.853ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116192025-04-30T06:35:58.853ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: fa 52 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .R..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:55598 req_id = 7df15047-783b-4e44-8a01-ec34fc23286e
116202025-04-30T06:35:58.853ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64082,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55598 req_id = 7df15047-783b-4e44-8a01-ec34fc23286e
116212025-04-30T06:35:58.854ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
116222025-04-30T06:35:58.854ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 64082, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:55598 req_id = 7df15047-783b-4e44-8a01-ec34fc23286e
116232025-04-30T06:35:58.854ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
116242025-04-30T06:35:58.854ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
116252025-04-30T06:35:58.854ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
116262025-04-30T06:35:58.856ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116272025-04-30T06:35:58.861ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116282025-04-30T06:35:58.861ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116292025-04-30T06:35:58.861ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116302025-04-30T06:35:58.862ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116312025-04-30T06:35:58.872ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bcdbece-d1b6-4466-8e9e-de23f5be0baa", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116322025-04-30T06:35:58.872ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
116332025-04-30T06:35:58.873ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116342025-04-30T06:35:58.873ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116352025-04-30T06:35:58.874ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116362025-04-30T06:35:58.875ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "598b5334-9916-42f3-8ba3-7099e71b763c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116372025-04-30T06:35:58.875ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
116382025-04-30T06:35:58.878ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce2663d7-c519-456a-8c6b-e794b85cc3a1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116392025-04-30T06:35:58.878ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
116402025-04-30T06:35:58.880ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "27ffb9f6-1fcd-4ece-bf3e-e646fb6fad1f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116412025-04-30T06:35:58.880ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
116422025-04-30T06:35:58.883ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "05269ede-0a88-48dc-8ac1-38faeabb2e05", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116432025-04-30T06:35:58.883ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
116442025-04-30T06:35:58.886ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "029dcabf-ae8e-41bc-8468-3e14b4199f9d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116452025-04-30T06:35:58.886ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
116462025-04-30T06:35:58.888ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "64c302b2-8929-44f4-8c35-d3e1d39ec058", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116472025-04-30T06:35:58.888ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
116482025-04-30T06:35:58.890ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "865e0db6-baff-4f90-8d22-cdabc96e49d2", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116492025-04-30T06:35:58.890ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
116502025-04-30T06:35:58.891ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0dd2d0d3-6543-45c3-8537-974ce2d45df4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116512025-04-30T06:35:58.891ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
116522025-04-30T06:35:58.893ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "26a21804-a1ec-489c-a208-9a40cb1ad356", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116532025-04-30T06:35:58.893ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
116542025-04-30T06:35:58.894ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5caa59c2-29bb-478a-aa63-a1ee6a8cd8c7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116552025-04-30T06:35:58.894ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
116562025-04-30T06:35:58.895ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bbb35d5a-bf73-442e-a0bd-ce109b37fe72", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116572025-04-30T06:35:58.895ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
116582025-04-30T06:35:58.896ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "49a0036c-3476-4306-a103-fbc92366ae51", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116592025-04-30T06:35:58.897ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
116602025-04-30T06:35:58.899ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6cf511f-fea1-4899-9d2e-1d38b139c14b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116612025-04-30T06:35:58.900ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
116622025-04-30T06:35:58.903ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
116632025-04-30T06:35:58.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "aa7b3152-0871-44b5-bdfd-43b4310597ad", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116642025-04-30T06:35:58.905ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
116652025-04-30T06:35:58.913ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcc2e27a-8d6b-4926-8a9a-d6ec83ee887e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116662025-04-30T06:35:58.913ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
116672025-04-30T06:35:58.915ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2510cfe-750f-4f7f-b1b0-1377a7730c77", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116682025-04-30T06:35:58.915ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
116692025-04-30T06:35:58.917ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5be203b3-df77-4a80-9705-5c9021021a5d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116702025-04-30T06:35:58.917ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
116712025-04-30T06:35:58.919ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "da0f5601-bf8b-4122-81d4-fc9de605a81c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116722025-04-30T06:35:58.919ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
116732025-04-30T06:35:58.920ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb514ba2-34fa-4b2a-b4aa-7a6609f83f93", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116742025-04-30T06:35:58.920ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
116752025-04-30T06:35:58.922ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e7534ec9-1341-4755-91c9-5d87fdcfcdc9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116762025-04-30T06:35:58.922ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
116772025-04-30T06:35:58.923ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dba49668-5ac6-433b-beab-b43ac72e70df", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116782025-04-30T06:35:58.923ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
116792025-04-30T06:35:58.924ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ab184faf-1b26-4b7d-8508-a07d60f7e5c2", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116802025-04-30T06:35:58.924ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
116812025-04-30T06:35:58.926ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cab28a92-5170-4131-84b0-7eada40a443e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116822025-04-30T06:35:58.926ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
116832025-04-30T06:35:58.948ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "80279ea1-55d3-4c93-a709-d6ecf795469e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116842025-04-30T06:35:58.948ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
116852025-04-30T06:35:58.950ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c7d7c29-43de-4217-af95-006c454c0a50", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116862025-04-30T06:35:58.950ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
116872025-04-30T06:35:58.952ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e00c625-bda3-4212-82dc-e206545c28a3", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116882025-04-30T06:35:58.953ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
116892025-04-30T06:35:58.954ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
116902025-04-30T06:35:58.954ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
116912025-04-30T06:35:58.954ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116922025-04-30T06:35:58.954ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
116932025-04-30T06:35:58.955ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
116942025-04-30T06:35:58.956ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fcda2d28-fb18-4aaa-b781-7cfa571be4bb", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
116952025-04-30T06:35:58.956ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
116962025-04-30T06:35:58.957ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
116972025-04-30T06:35:58.962ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
116982025-04-30T06:35:58.962ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
116992025-04-30T06:35:58.962ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117002025-04-30T06:35:58.963ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117012025-04-30T06:35:58.964ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "310b4d64-9f59-4a5d-9127-3c2aee13f36f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117022025-04-30T06:35:58.964ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
117032025-04-30T06:35:58.965ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117042025-04-30T06:35:58.965ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117052025-04-30T06:35:58.966ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117062025-04-30T06:35:58.969ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c387f32d-3524-4864-8f66-9e6ca9b9109f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117072025-04-30T06:35:58.969ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
117082025-04-30T06:35:58.971ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "92539809-286c-4e59-b1bc-cb650446b40e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117092025-04-30T06:35:58.971ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
117102025-04-30T06:35:58.973ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "31457ddb-88b4-4d44-8798-df3621647b99", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117112025-04-30T06:35:58.973ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
117122025-04-30T06:35:58.974ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bd80cdc-4e89-4d01-a69f-bf4686abb52e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117132025-04-30T06:35:58.974ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
117142025-04-30T06:35:58.976ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "36cb777c-37d6-4d60-ae8b-7172ab44488c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117152025-04-30T06:35:58.976ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
117162025-04-30T06:35:58.978ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6ae2410-1c87-473e-9056-5ee2cd50c9f6", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117172025-04-30T06:35:58.978ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
117182025-04-30T06:35:58.980ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cf7dfd52-8903-44ad-b469-f3dfceb9c67a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117192025-04-30T06:35:58.980ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
117202025-04-30T06:35:58.982ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81f4c0f7-b647-4542-b78b-2af47fdf83bf", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117212025-04-30T06:35:58.982ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
117222025-04-30T06:35:58.985ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c7028b7-d983-4548-9f6b-ec5edbdde6cc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117232025-04-30T06:35:58.986ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
117242025-04-30T06:35:58.989ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "067f99ea-9c2f-4039-94ef-1fd5dca4655c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117252025-04-30T06:35:58.989ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
117262025-04-30T06:35:58.991ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b7fd4861-ef7a-4f4f-ac5c-5f57c4b12576", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117272025-04-30T06:35:58.991ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
117282025-04-30T06:35:58.993ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb571eaa-22c7-487e-9841-f7ee3f3154bc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117292025-04-30T06:35:58.994ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
117302025-04-30T06:35:58.994ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "106de9c7-09ac-44e0-ae3b-3148432140f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117312025-04-30T06:35:58.995ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
117322025-04-30T06:35:58.996ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3796660-33ca-48c0-b496-df9949d9bba5", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117332025-04-30T06:35:58.996ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
117342025-04-30T06:35:58.999ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a61ed9f6-7b98-486b-a5cc-2a97f4424063", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117352025-04-30T06:35:58.999ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
117362025-04-30T06:35:59.002ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
117372025-04-30T06:35:59.007ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "344d4103-e771-4036-8ab5-7161317fa39f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:58 GMT"} })
117382025-04-30T06:35:59.007ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117392025-04-30T06:35:59.008ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117402025-04-30T06:35:59.008ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117412025-04-30T06:35:59.008ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117422025-04-30T06:35:59.009ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117432025-04-30T06:35:59.013ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117442025-04-30T06:35:59.018ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117452025-04-30T06:35:59.018ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
117462025-04-30T06:35:59.018ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117472025-04-30T06:35:59.018ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117482025-04-30T06:35:59.018ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117492025-04-30T06:35:59.019ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117502025-04-30T06:35:59.022ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117512025-04-30T06:35:59.027ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
117522025-04-30T06:35:59.028ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117532025-04-30T06:35:59.028ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117542025-04-30T06:35:59.029ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117552025-04-30T06:35:59.086ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
117562025-04-30T06:35:59.087ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
117572025-04-30T06:35:59.088ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
117582025-04-30T06:35:59.088ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117592025-04-30T06:35:59.088ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117602025-04-30T06:35:59.088ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117612025-04-30T06:35:59.089ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117622025-04-30T06:35:59.089ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:54405 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
117632025-04-30T06:35:59.092ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:54405 req_id = 48436b1c-e879-4236-88ca-353f421db0e8 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117642025-04-30T06:35:59.092ZINFOtest_metrics (dropshot): request completed latency_us = 621 local_addr = [::1]:56237 method = GET remote_addr = [::1]:54405 req_id = 48436b1c-e879-4236-88ca-353f421db0e8 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
117652025-04-30T06:35:59.093ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117662025-04-30T06:35:59.106ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
117672025-04-30T06:35:59.106ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117682025-04-30T06:35:59.106ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117692025-04-30T06:35:59.107ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117702025-04-30T06:35:59.108ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "48436b1c-e879-4236-88ca-353f421db0e8", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
117712025-04-30T06:35:59.112ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117722025-04-30T06:35:59.112ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
117732025-04-30T06:35:59.112ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
117742025-04-30T06:35:59.114ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117752025-04-30T06:35:59.114ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117762025-04-30T06:35:59.115ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117772025-04-30T06:35:59.115ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117782025-04-30T06:35:59.115ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117792025-04-30T06:35:59.120ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
117802025-04-30T06:35:59.122ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
117812025-04-30T06:35:59.124ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
117822025-04-30T06:35:59.125ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117832025-04-30T06:35:59.131ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117842025-04-30T06:35:59.132ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117852025-04-30T06:35:59.132ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117862025-04-30T06:35:59.134ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117872025-04-30T06:35:59.179ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
117882025-04-30T06:35:59.180ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
117892025-04-30T06:35:59.180ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 393.600343ms iteration = 41 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
117902025-04-30T06:35:59.180ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 42 reason = Signaled
117912025-04-30T06:35:59.180ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
117922025-04-30T06:35:59.181ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
117932025-04-30T06:35:59.181ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
117942025-04-30T06:35:59.181ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
117952025-04-30T06:35:59.182ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
117962025-04-30T06:35:59.183ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
117972025-04-30T06:35:59.185ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
117982025-04-30T06:35:59.189ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
117992025-04-30T06:35:59.189ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
118002025-04-30T06:35:59.190ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
118012025-04-30T06:35:59.190ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
118022025-04-30T06:35:59.193ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
118032025-04-30T06:35:59.194ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
118042025-04-30T06:35:59.195ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: a4 7b 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .{..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52440 req_id = b72674fb-2f3e-4a26-a5e1-9f3b780da817
118052025-04-30T06:35:59.195ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42107,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52440 req_id = b72674fb-2f3e-4a26-a5e1-9f3b780da817
118062025-04-30T06:35:59.195ZDEBGtest_metrics (store): query key key = _mgs._tcp
118072025-04-30T06:35:59.195ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118082025-04-30T06:35:59.195ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52440 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = b72674fb-2f3e-4a26-a5e1-9f3b780da817
118092025-04-30T06:35:59.196ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3823, tv_nsec: 995264146 } })
118102025-04-30T06:35:59.196ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
118112025-04-30T06:35:59.196ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: bd 0b 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58896 req_id = 24bd193e-7877-4510-a6d4-447c023be688
118122025-04-30T06:35:59.196ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 48395,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58896 req_id = 24bd193e-7877-4510-a6d4-447c023be688
118132025-04-30T06:35:59.197ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
118142025-04-30T06:35:59.197ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58896 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 24bd193e-7877-4510-a6d4-447c023be688
118152025-04-30T06:35:59.258ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
118162025-04-30T06:35:59.258ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118172025-04-30T06:35:59.258ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118182025-04-30T06:35:59.259ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118192025-04-30T06:35:59.259ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118202025-04-30T06:35:59.260ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118212025-04-30T06:35:59.261ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 54 94 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c T............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59367 req_id = f7f7a545-68c5-4451-b756-a299867be275
118222025-04-30T06:35:59.261ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 21652,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59367 req_id = f7f7a545-68c5-4451-b756-a299867be275
118232025-04-30T06:35:59.261ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
118242025-04-30T06:35:59.261ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 21652, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59367 req_id = f7f7a545-68c5-4451-b756-a299867be275
118252025-04-30T06:35:59.262ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
118262025-04-30T06:35:59.262ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
118272025-04-30T06:35:59.262ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
118282025-04-30T06:35:59.263ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
118292025-04-30T06:35:59.269ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
118302025-04-30T06:35:59.269ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118312025-04-30T06:35:59.269ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118322025-04-30T06:35:59.270ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118332025-04-30T06:35:59.272ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8a32a7e-8fdd-4598-b9cf-9f11b79db9e3", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118342025-04-30T06:35:59.272ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
118352025-04-30T06:35:59.272ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118362025-04-30T06:35:59.273ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118372025-04-30T06:35:59.273ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
118382025-04-30T06:35:59.274ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9033e01f-b02b-46dd-80c5-39ea1d4e32cf", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118392025-04-30T06:35:59.275ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
118402025-04-30T06:35:59.279ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "55ee6ca7-01db-4a90-b7b2-c749d2b4c576", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118412025-04-30T06:35:59.280ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
118422025-04-30T06:35:59.282ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af6a5d21-3418-401b-9d7e-87446602bc58", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118432025-04-30T06:35:59.282ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
118442025-04-30T06:35:59.284ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4422258f-c7f2-4f35-a398-88d82448d01f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118452025-04-30T06:35:59.284ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
118462025-04-30T06:35:59.287ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "75f2b8f0-7463-44b8-a077-b218b1bdfe7b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118472025-04-30T06:35:59.287ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
118482025-04-30T06:35:59.289ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9d562e8f-fb8c-44a9-a286-5cea5976a6ea", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118492025-04-30T06:35:59.289ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
118502025-04-30T06:35:59.293ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "397c3c33-2801-44c7-8854-d6ee5c37b77f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118512025-04-30T06:35:59.293ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
118522025-04-30T06:35:59.295ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
118532025-04-30T06:35:59.296ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
118542025-04-30T06:35:59.297ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 51f6eca0-fdbc-47cb-895d-463a5cb2f7dc uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
118552025-04-30T06:35:59.297ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "defceb4d-62f6-405c-a31c-ac7d5b6c83aa", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118562025-04-30T06:35:59.298ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
118572025-04-30T06:35:59.298ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 51f6eca0-fdbc-47cb-895d-463a5cb2f7dc resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
118582025-04-30T06:35:59.298ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 51f6eca0-fdbc-47cb-895d-463a5cb2f7dc roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
118592025-04-30T06:35:59.299ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 51f6eca0-fdbc-47cb-895d-463a5cb2f7dc resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
118602025-04-30T06:35:59.301ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bd3c2127-5f5f-4f6a-8557-bef67eccea9f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118612025-04-30T06:35:59.301ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
118622025-04-30T06:35:59.304ZINFOtest_metrics (dropshot_internal): request completed latency_us = 7218 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 51f6eca0-fdbc-47cb-895d-463a5cb2f7dc response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
118632025-04-30T06:35:59.305ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
118642025-04-30T06:35:59.306ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "51f6eca0-fdbc-47cb-895d-463a5cb2f7dc", "content-length": "405", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118652025-04-30T06:35:59.306ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
118662025-04-30T06:35:59.307ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 74cd45c9-bea7-41d8-8255-17605e7c986f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
118672025-04-30T06:35:59.308ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 74cd45c9-bea7-41d8-8255-17605e7c986f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
118682025-04-30T06:35:59.308ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 74cd45c9-bea7-41d8-8255-17605e7c986f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
118692025-04-30T06:35:59.308ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 74cd45c9-bea7-41d8-8255-17605e7c986f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
118702025-04-30T06:35:59.309ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4da64578-ace4-4b80-a14c-82f1d28f42c6", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118712025-04-30T06:35:59.309ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
118722025-04-30T06:35:59.310ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84da2db2-2911-4ad3-bd16-921943f24cdd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118732025-04-30T06:35:59.310ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
118742025-04-30T06:35:59.311ZINFOtest_metrics (dropshot_internal): request completed latency_us = 4166 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 74cd45c9-bea7-41d8-8255-17605e7c986f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
118752025-04-30T06:35:59.312ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "74cd45c9-bea7-41d8-8255-17605e7c986f", "content-length": "29", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118762025-04-30T06:35:59.312ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
118772025-04-30T06:35:59.312ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
118782025-04-30T06:35:59.312ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
118792025-04-30T06:35:59.313ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3abbe8f4-fd61-4258-8861-b784e020cfa0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118802025-04-30T06:35:59.313ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
118812025-04-30T06:35:59.315ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e3c8427-761d-48a2-9484-0a44588d2d0e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118822025-04-30T06:35:59.315ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
118832025-04-30T06:35:59.317ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d8d4b01-bcf6-4cb4-98c0-0bdfc178d8e1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118842025-04-30T06:35:59.317ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
118852025-04-30T06:35:59.319ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "59e78bb9-dac6-4735-8102-67b28c8678e8", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118862025-04-30T06:35:59.319ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
118872025-04-30T06:35:59.323ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e4b94d2-513a-4395-bc14-dd5114621a31", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118882025-04-30T06:35:59.323ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
118892025-04-30T06:35:59.326ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a462f77a-8231-424d-95d9-6f25d134ad6f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118902025-04-30T06:35:59.327ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
118912025-04-30T06:35:59.349ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "522cd17c-bd0a-4d3d-baba-8eb146b79a7b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118922025-04-30T06:35:59.349ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
118932025-04-30T06:35:59.357ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b39280e-5ee9-4244-9a12-9a5d13b612bd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
118942025-04-30T06:35:59.357ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
118952025-04-30T06:35:59.359ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
118962025-04-30T06:35:59.359ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
118972025-04-30T06:35:59.359ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
118982025-04-30T06:35:59.359ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
118992025-04-30T06:35:59.360ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119002025-04-30T06:35:59.372ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "851a6c77-83ba-476a-ba06-c55e9b3f1a08", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119012025-04-30T06:35:59.373ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
119022025-04-30T06:35:59.373ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119032025-04-30T06:35:59.380ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119042025-04-30T06:35:59.380ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119052025-04-30T06:35:59.380ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119062025-04-30T06:35:59.381ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119072025-04-30T06:35:59.382ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "42b2bd55-0996-4fef-99ff-8497ff0b27d8", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119082025-04-30T06:35:59.382ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
119092025-04-30T06:35:59.383ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c26fc3e-1f88-4c8b-87fa-e15021ed34e7", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119102025-04-30T06:35:59.384ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
119112025-04-30T06:35:59.384ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119122025-04-30T06:35:59.384ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119132025-04-30T06:35:59.384ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119142025-04-30T06:35:59.385ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "56e7b50c-f489-42ea-89a8-256742105093", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119152025-04-30T06:35:59.386ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
119162025-04-30T06:35:59.388ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d789611-0343-4801-81d5-4c6e1008c8ec", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119172025-04-30T06:35:59.388ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
119182025-04-30T06:35:59.390ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3308b0b1-feb1-4b85-a294-4fc5eb8002fe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119192025-04-30T06:35:59.390ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
119202025-04-30T06:35:59.392ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b72f5fb8-e0fe-465c-87ff-6b4babe42dcc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119212025-04-30T06:35:59.392ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
119222025-04-30T06:35:59.395ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a3df7673-3679-402c-985b-61a2a90b5c1f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119232025-04-30T06:35:59.395ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
119242025-04-30T06:35:59.397ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "47522f8f-42ef-44be-84b7-aeb559982907", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119252025-04-30T06:35:59.397ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
119262025-04-30T06:35:59.400ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b64ac4dd-b502-44e7-9a72-f0732bee68da", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119272025-04-30T06:35:59.401ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
119282025-04-30T06:35:59.402ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e752384-e1f9-4719-93b8-8bf43b97b468", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119292025-04-30T06:35:59.402ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
119302025-04-30T06:35:59.404ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37fb3910-43be-40d2-b32a-977ea514482f", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119312025-04-30T06:35:59.405ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
119322025-04-30T06:35:59.408ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e36c0b9a-c02f-430a-8e5e-a1839d205457", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119332025-04-30T06:35:59.408ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
119342025-04-30T06:35:59.409ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
119352025-04-30T06:35:59.409ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
119362025-04-30T06:35:59.410ZTRACtest_metrics (dropshot): incoming request local_addr = 127.0.0.1:65131 method = GET remote_addr = 127.0.0.1:48235 req_id = 2157fbb5-299d-40b3-834e-12a045815de7 uri = /588db90d-5640-4be3-9102-2ebf0af0c9a3
119372025-04-30T06:35:59.410ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ffb3206-bd3b-4055-a84e-b82cc2ed2d33", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119382025-04-30T06:35:59.411ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
119392025-04-30T06:35:59.427ZINFOtest_metrics (dropshot): request completed file = /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/dropshot-0.16.0/src/server.rs:867 latency_us = 16995 local_addr = 127.0.0.1:65131 method = GET remote_addr = 127.0.0.1:48235 req_id = 2157fbb5-299d-40b3-834e-12a045815de7 response_code = 200 uri = /588db90d-5640-4be3-9102-2ebf0af0c9a3
119402025-04-30T06:35:59.436ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
119412025-04-30T06:35:59.450ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 5 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
119422025-04-30T06:35:59.450ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
119432025-04-30T06:35:59.450ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
119442025-04-30T06:35:59.451ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
119452025-04-30T06:35:59.454ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9a006339-c8ae-48f0-a3e7-26ea8d8b6211", "content-length": "723", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119462025-04-30T06:35:59.454ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
119472025-04-30T06:35:59.459ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a3f5964-7daa-4e25-bf6c-37e28549bb44", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119482025-04-30T06:35:59.459ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
119492025-04-30T06:35:59.463ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "57a5437d-5623-4f86-81dd-37f2ee006f9f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119502025-04-30T06:35:59.463ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
119512025-04-30T06:35:59.467ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f80bab9-adfa-4290-bf21-fc647426c58b", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119522025-04-30T06:35:59.467ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
119532025-04-30T06:35:59.471ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6104837-b1bb-404d-8f77-1071dedcb33e", "content-length": "169", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119542025-04-30T06:35:59.471ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
119552025-04-30T06:35:59.474ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "09715cd2-7e8b-4b74-9f6e-4a7ef314a972", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119562025-04-30T06:35:59.475ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
119572025-04-30T06:35:59.478ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dc3dcb8d-e79a-4c41-9ca4-9aa1bd957710", "content-length": "178", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119582025-04-30T06:35:59.478ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
119592025-04-30T06:35:59.479ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5450a5c8-901b-48d2-b983-43cd09e06d36", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119602025-04-30T06:35:59.479ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
119612025-04-30T06:35:59.481ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8e4b5dc-5fa8-43f4-b628-45e707b0e8a2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119622025-04-30T06:35:59.481ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
119632025-04-30T06:35:59.482ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44cdf6e8-2ee1-4e41-9e7a-24da41b835b2", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119642025-04-30T06:35:59.482ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
119652025-04-30T06:35:59.483ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3ab199ea-b9c0-4859-874a-bb80d58e6c62", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
119662025-04-30T06:35:59.483ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119672025-04-30T06:35:59.484ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119682025-04-30T06:35:59.484ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119692025-04-30T06:35:59.484ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119702025-04-30T06:35:59.485ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119712025-04-30T06:35:59.488ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119722025-04-30T06:35:59.492ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119732025-04-30T06:35:59.492ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
119742025-04-30T06:35:59.492ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119752025-04-30T06:35:59.492ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119762025-04-30T06:35:59.492ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119772025-04-30T06:35:59.492ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119782025-04-30T06:35:59.502ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
119792025-04-30T06:35:59.502ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
119802025-04-30T06:35:59.502ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119812025-04-30T06:35:59.502ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119822025-04-30T06:35:59.503ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119832025-04-30T06:35:59.504ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119842025-04-30T06:35:59.508ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
119852025-04-30T06:35:59.508ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
119862025-04-30T06:35:59.508ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
119872025-04-30T06:35:59.509ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
119882025-04-30T06:35:59.512ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
119892025-04-30T06:35:59.534ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
119902025-04-30T06:35:59.534ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
119912025-04-30T06:35:59.534ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
119922025-04-30T06:35:59.535ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
119932025-04-30T06:35:59.587ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
119942025-04-30T06:35:59.587ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
119952025-04-30T06:35:59.589ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:41266 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
119962025-04-30T06:35:59.590ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
119972025-04-30T06:35:59.590ZDEBGtest_metrics (results-sink): inserting 8 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119982025-04-30T06:35:59.590ZDEBGtest_metrics (results-sink-cluster): inserting 8 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
119992025-04-30T06:35:59.590ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120002025-04-30T06:35:59.590ZDEBGtest_metrics (clickhouse-client): unrolling 8 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
120012025-04-30T06:35:59.599ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 16 sql = INSERT INTO oximeter.fields_string FORMAT Native
120022025-04-30T06:35:59.600ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
120032025-04-30T06:35:59.601ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
120042025-04-30T06:35:59.601ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
120052025-04-30T06:35:59.605ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:41266 req_id = 9b251cba-8947-4d01-a193-bc7b20183c4d sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120062025-04-30T06:35:59.605ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
120072025-04-30T06:35:59.609ZINFOtest_metrics (dropshot): request completed latency_us = 3992 local_addr = [::1]:56237 method = GET remote_addr = [::1]:41266 req_id = 9b251cba-8947-4d01-a193-bc7b20183c4d response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
120082025-04-30T06:35:59.609ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 16 table_name = fields_string
120092025-04-30T06:35:59.610ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 8 sql = INSERT INTO oximeter.fields_u16 FORMAT Native
120102025-04-30T06:35:59.611ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120112025-04-30T06:35:59.613ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120122025-04-30T06:35:59.613ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120132025-04-30T06:35:59.613ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
120142025-04-30T06:35:59.613ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b251cba-8947-4d01-a193-bc7b20183c4d", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
120152025-04-30T06:35:59.624ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120162025-04-30T06:35:59.624ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
120172025-04-30T06:35:59.624ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
120182025-04-30T06:35:59.628ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120192025-04-30T06:35:59.628ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120202025-04-30T06:35:59.628ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120212025-04-30T06:35:59.628ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120222025-04-30T06:35:59.629ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120232025-04-30T06:35:59.633ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 7f 26 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .&..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53441 req_id = 3930029d-1dc1-4385-b357-02e5323b2afd
120242025-04-30T06:35:59.633ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 32550,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53441 req_id = 3930029d-1dc1-4385-b357-02e5323b2afd
120252025-04-30T06:35:59.634ZDEBGtest_metrics (store): query key key = _dendrite._tcp
120262025-04-30T06:35:59.634ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120272025-04-30T06:35:59.634ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53441 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 3930029d-1dc1-4385-b357-02e5323b2afd
120282025-04-30T06:35:59.637ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 8 table_name = fields_u16
120292025-04-30T06:35:59.637ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 8 sql = INSERT INTO oximeter.fields_uuid FORMAT Native
120302025-04-30T06:35:59.639ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3824, tv_nsec: 438091540 } })
120312025-04-30T06:35:59.639ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
120322025-04-30T06:35:59.639ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120332025-04-30T06:35:59.644ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120342025-04-30T06:35:59.645ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120352025-04-30T06:35:59.645ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120362025-04-30T06:35:59.646ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120372025-04-30T06:35:59.650ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1a 77 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .w..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59274 req_id = f979b4a0-29a1-4046-921f-d5483e9fc704
120382025-04-30T06:35:59.650ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6775,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59274 req_id = f979b4a0-29a1-4046-921f-d5483e9fc704
120392025-04-30T06:35:59.650ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120402025-04-30T06:35:59.650ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59274 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f979b4a0-29a1-4046-921f-d5483e9fc704
120412025-04-30T06:35:59.650ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
120422025-04-30T06:35:59.709ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
120432025-04-30T06:35:59.710ZDEBGtest_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
120442025-04-30T06:35:59.710ZDEBGtest_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
120452025-04-30T06:35:59.710ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
120462025-04-30T06:35:59.713ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
120472025-04-30T06:35:59.713ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 8 table_name = fields_uuid
120482025-04-30T06:35:59.713ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 19 n_rows = 8 sql = INSERT INTO oximeter.measurements_histogramu64 FORMAT Native
120492025-04-30T06:35:59.715ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
120502025-04-30T06:35:59.716ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
120512025-04-30T06:35:59.717ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 90 e5 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:52950 req_id = 07cfabf9-8ddf-40d1-bae0-5f2e46c5024d
120522025-04-30T06:35:59.718ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37093,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52950 req_id = 07cfabf9-8ddf-40d1-bae0-5f2e46c5024d
120532025-04-30T06:35:59.718ZDEBGtest_metrics (store): query key key = _dendrite._tcp
120542025-04-30T06:35:59.718ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120552025-04-30T06:35:59.718ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52950 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 07cfabf9-8ddf-40d1-bae0-5f2e46c5024d
120562025-04-30T06:35:59.719ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3824, tv_nsec: 518012237 } })
120572025-04-30T06:35:59.720ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
120582025-04-30T06:35:59.720ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 8 table_name = measurements_histogramu64
120592025-04-30T06:35:59.721ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
120602025-04-30T06:35:59.722ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1b d1 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57598 req_id = b98f6a93-f83a-4989-9376-cda44d8fa476
120612025-04-30T06:35:59.722ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7121,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57598 req_id = b98f6a93-f83a-4989-9376-cda44d8fa476
120622025-04-30T06:35:59.722ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120632025-04-30T06:35:59.722ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57598 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = b98f6a93-f83a-4989-9376-cda44d8fa476
120642025-04-30T06:35:59.722ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
120652025-04-30T06:35:59.771ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
120662025-04-30T06:35:59.771ZDEBGtest_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
120672025-04-30T06:35:59.772ZDEBGtest_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
120682025-04-30T06:35:59.772ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
120692025-04-30T06:35:59.774ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
120702025-04-30T06:35:59.792ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
120712025-04-30T06:35:59.792ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
120722025-04-30T06:35:59.793ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 612.420838ms iteration = 42 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
120732025-04-30T06:35:59.793ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 43 reason = Signaled
120742025-04-30T06:35:59.793ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
120752025-04-30T06:35:59.793ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
120762025-04-30T06:35:59.794ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120772025-04-30T06:35:59.794ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120782025-04-30T06:35:59.795ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120792025-04-30T06:35:59.799ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
120802025-04-30T06:35:59.800ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
120812025-04-30T06:35:59.808ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
120822025-04-30T06:35:59.808ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
120832025-04-30T06:35:59.808ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
120842025-04-30T06:35:59.809ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
120852025-04-30T06:35:59.828ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
120862025-04-30T06:35:59.828ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
120872025-04-30T06:35:59.834ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b4 21 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .!..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:65150 req_id = da68e50b-7c7c-4f1c-afaf-331c6c310044
120882025-04-30T06:35:59.836ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46113,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:65150 req_id = da68e50b-7c7c-4f1c-afaf-331c6c310044
120892025-04-30T06:35:59.836ZDEBGtest_metrics (store): query key key = _mgs._tcp
120902025-04-30T06:35:59.836ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120912025-04-30T06:35:59.836ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:65150 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = da68e50b-7c7c-4f1c-afaf-331c6c310044
120922025-04-30T06:35:59.845ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3824, tv_nsec: 644569323 } })
120932025-04-30T06:35:59.846ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
120942025-04-30T06:35:59.846ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
120952025-04-30T06:35:59.847ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 98 4a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .J..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:49796 req_id = 8ff0e357-69af-442e-8e67-e1051f4e18e3
120962025-04-30T06:35:59.847ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38986,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49796 req_id = 8ff0e357-69af-442e-8e67-e1051f4e18e3
120972025-04-30T06:35:59.847ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
120982025-04-30T06:35:59.847ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:49796 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 8ff0e357-69af-442e-8e67-e1051f4e18e3
120992025-04-30T06:35:59.910ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
121002025-04-30T06:35:59.910ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: df 15 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63524 req_id = eb1b1d67-a9b8-46b7-a1c5-9513b5a9b6e0
121012025-04-30T06:35:59.910ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57109,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63524 req_id = eb1b1d67-a9b8-46b7-a1c5-9513b5a9b6e0
121022025-04-30T06:35:59.911ZDEBGtest_metrics (store): query key key = _dendrite._tcp
121032025-04-30T06:35:59.911ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121042025-04-30T06:35:59.911ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63524 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = eb1b1d67-a9b8-46b7-a1c5-9513b5a9b6e0
121052025-04-30T06:35:59.911ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121062025-04-30T06:35:59.911ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121072025-04-30T06:35:59.911ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121082025-04-30T06:35:59.911ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121092025-04-30T06:35:59.912ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121102025-04-30T06:35:59.912ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3824, tv_nsec: 711736499 } })
121112025-04-30T06:35:59.912ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
121122025-04-30T06:35:59.913ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1e 52 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .R..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:51198 req_id = 4b8fd998-0886-4c32-beed-8048719d4ff6
121132025-04-30T06:35:59.913ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7762,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51198 req_id = 4b8fd998-0886-4c32-beed-8048719d4ff6
121142025-04-30T06:35:59.913ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
121152025-04-30T06:35:59.913ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7762, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:51198 req_id = 4b8fd998-0886-4c32-beed-8048719d4ff6
121162025-04-30T06:35:59.914ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
121172025-04-30T06:35:59.914ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
121182025-04-30T06:35:59.914ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
121192025-04-30T06:35:59.914ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 76 ac 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e v...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63859 req_id = 1b724f35-7369-4f91-8850-13f84b500f09
121202025-04-30T06:35:59.914ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 30380,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63859 req_id = 1b724f35-7369-4f91-8850-13f84b500f09
121212025-04-30T06:35:59.914ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
121222025-04-30T06:35:59.914ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63859 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 1b724f35-7369-4f91-8850-13f84b500f09
121232025-04-30T06:35:59.915ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
121242025-04-30T06:35:59.965ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
121252025-04-30T06:35:59.965ZDEBGtest_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
121262025-04-30T06:35:59.965ZDEBGtest_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
121272025-04-30T06:35:59.965ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
121282025-04-30T06:35:59.966ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
121292025-04-30T06:35:59.967ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121302025-04-30T06:35:59.974ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121312025-04-30T06:35:59.974ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121322025-04-30T06:35:59.974ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121332025-04-30T06:35:59.975ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121342025-04-30T06:35:59.983ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121352025-04-30T06:35:59.983ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121362025-04-30T06:35:59.984ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121372025-04-30T06:35:59.984ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "811f140b-47d5-4b12-9e23-c5d6cd51d672", "content-length": "835", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121382025-04-30T06:35:59.985ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
121392025-04-30T06:35:59.987ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2503eb3f-fd0a-4a1d-960a-3c76faeeeb9d", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121402025-04-30T06:35:59.987ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
121412025-04-30T06:35:59.997ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "087c9750-88b9-4b54-9a5d-53cb6866d667", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121422025-04-30T06:35:59.997ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
121432025-04-30T06:35:59.999ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "77fdf644-27d8-4e70-9043-6e3a3eaeb323", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121442025-04-30T06:36:00.000ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
121452025-04-30T06:36:00.006ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "6222f3a2-c084-417d-9d0d-70d7763cb04a", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121462025-04-30T06:36:00.011ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
121472025-04-30T06:36:00.011ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4945ea42-7bb8-457b-8cdb-61ac8d0848e9", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121482025-04-30T06:36:00.011ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
121492025-04-30T06:36:00.014ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a6954d2b-7843-43f0-ac48-dabdd53ba7f6", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121502025-04-30T06:36:00.014ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
121512025-04-30T06:36:00.019ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32fcd714-e7b7-4494-ad2c-25440741b414", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121522025-04-30T06:36:00.019ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
121532025-04-30T06:36:00.020ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "147a4eaf-e1ec-49f8-a54c-fb40d88353fc", "content-length": "702", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121542025-04-30T06:36:00.020ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
121552025-04-30T06:36:00.038ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d2af856-8099-4b61-8da1-2fbf7f3af705", "content-length": "727", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121562025-04-30T06:36:00.038ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
121572025-04-30T06:36:00.042ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d09df2e3-aa52-4af5-ad58-917299d1cfd3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121582025-04-30T06:36:00.042ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
121592025-04-30T06:36:00.045ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9daf628-d446-4b61-a1f9-87e6661d2bf0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121602025-04-30T06:36:00.045ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
121612025-04-30T06:36:00.058ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "367cd69d-7e6f-4f99-b67e-b09793f5a6f0", "content-length": "734", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121622025-04-30T06:36:00.058ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
121632025-04-30T06:36:00.065ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
121642025-04-30T06:36:00.069ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "66bfb945-4214-48a3-8627-20904a0531dc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121652025-04-30T06:36:00.070ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
121662025-04-30T06:36:00.078ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb884453-844d-4a0e-8478-16377255681e", "content-length": "111", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121672025-04-30T06:36:00.079ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
121682025-04-30T06:36:00.083ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e578c72f-1177-4e07-b263-2197bd0b1999", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121692025-04-30T06:36:00.084ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
121702025-04-30T06:36:00.086ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "915130a0-7fa2-4498-8e47-e17523a2d3e4", "content-length": "170", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121712025-04-30T06:36:00.086ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
121722025-04-30T06:36:00.089ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ada6f5f-f7ad-4a0c-8073-bb19acf6e835", "content-length": "179", "date": "Wed, 30 Apr 2025 06:35:59 GMT"} })
121732025-04-30T06:36:00.089ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
121742025-04-30T06:36:00.099ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b6f36f2-3f29-47a1-b795-64182d1d12bf", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
121752025-04-30T06:36:00.100ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
121762025-04-30T06:36:00.102ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "240a1366-f17e-49e2-bc09-a27fcd226449", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
121772025-04-30T06:36:00.102ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
121782025-04-30T06:36:00.105ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7e43b860-1278-46a2-86b3-9a15d24f8c2d", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
121792025-04-30T06:36:00.105ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
121802025-04-30T06:36:00.109ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2a3ea720-a240-46dd-8266-ec1253843bf3", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
121812025-04-30T06:36:00.109ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
121822025-04-30T06:36:00.112ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "00b3f00e-f85c-4c58-8780-f1fbd303020a", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
121832025-04-30T06:36:00.113ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
121842025-04-30T06:36:00.116ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
121852025-04-30T06:36:00.116ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
121862025-04-30T06:36:00.116ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121872025-04-30T06:36:00.116ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121882025-04-30T06:36:00.117ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121892025-04-30T06:36:00.118ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ca2d7e1d-0184-4bb9-9201-cc45fddf5664", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
121902025-04-30T06:36:00.118ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
121912025-04-30T06:36:00.129ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
121922025-04-30T06:36:00.161ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
121932025-04-30T06:36:00.161ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121942025-04-30T06:36:00.161ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121952025-04-30T06:36:00.161ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121962025-04-30T06:36:00.166ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
121972025-04-30T06:36:00.166ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
121982025-04-30T06:36:00.176ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
121992025-04-30T06:36:00.215ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "470de00e-e670-4d11-8fb8-468a706d95a1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122002025-04-30T06:36:00.217ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
122012025-04-30T06:36:00.238ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8bca5e4-8a55-4016-89d0-9fcff8dc3127", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122022025-04-30T06:36:00.238ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
122032025-04-30T06:36:00.289ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3c4d2794-f859-4626-8a73-7bc3971f3a42", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122042025-04-30T06:36:00.289ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
122052025-04-30T06:36:00.311ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "525019ac-cb18-450f-b7b0-82659add5133", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122062025-04-30T06:36:00.312ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
122072025-04-30T06:36:00.323ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
122082025-04-30T06:36:00.328ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a630583e-31d7-4955-a9fa-526d9b5f49e5", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122092025-04-30T06:36:00.328ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
122102025-04-30T06:36:00.369ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fdc0c8b3-a4a6-49c3-bd84-6443b9f4b851", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122112025-04-30T06:36:00.369ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
122122025-04-30T06:36:00.374ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122132025-04-30T06:36:00.374ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122142025-04-30T06:36:00.374ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122152025-04-30T06:36:00.374ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122162025-04-30T06:36:00.375ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122172025-04-30T06:36:00.376ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce2a5578-227c-489d-a253-a2aff465ad1f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122182025-04-30T06:36:00.376ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
122192025-04-30T06:36:00.378ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122202025-04-30T06:36:00.383ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122212025-04-30T06:36:00.383ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122222025-04-30T06:36:00.383ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122232025-04-30T06:36:00.383ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122242025-04-30T06:36:00.403ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8533453-3831-43fc-a942-688e831f3966", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122252025-04-30T06:36:00.403ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
122262025-04-30T06:36:00.405ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c991ead-9696-4d0b-a438-ea52c4b93db0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122272025-04-30T06:36:00.405ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
122282025-04-30T06:36:00.406ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122292025-04-30T06:36:00.406ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122302025-04-30T06:36:00.409ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122312025-04-30T06:36:00.411ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a4eeeb5-6079-4f75-ae9f-f86b3d697874", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122322025-04-30T06:36:00.411ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
122332025-04-30T06:36:00.414ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d58e3efc-c207-4bda-a571-9a3b33c36988", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122342025-04-30T06:36:00.422ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
122352025-04-30T06:36:00.422ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "38a87677-87b4-49c8-9661-678d216a4128", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122362025-04-30T06:36:00.426ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
122372025-04-30T06:36:00.442ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "78cea2f7-b3e4-4cc2-b650-c0e7107503d5", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122382025-04-30T06:36:00.442ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
122392025-04-30T06:36:00.450ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9dad3239-f063-4ec8-882e-e6404182cf37", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122402025-04-30T06:36:00.450ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
122412025-04-30T06:36:00.457ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9bfae2c7-4f53-449a-8e6e-3111f87c3a7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122422025-04-30T06:36:00.458ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
122432025-04-30T06:36:00.474ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "74b35ad5-d641-491b-ad2f-4ab02d1174cf", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122442025-04-30T06:36:00.474ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
122452025-04-30T06:36:00.487ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "91b40305-f60e-4ed4-8621-a871d53432fb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122462025-04-30T06:36:00.487ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
122472025-04-30T06:36:00.498ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
122482025-04-30T06:36:00.499ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "35aa5948-585f-430a-a9a2-45639c013164", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122492025-04-30T06:36:00.499ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
122502025-04-30T06:36:00.506ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "93587ee8-aa3d-4bdd-874c-5ec73a6c7829", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122512025-04-30T06:36:00.507ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
122522025-04-30T06:36:00.508ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "66aded0b-7cfe-4012-aae3-38d4f81dbe1e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122532025-04-30T06:36:00.508ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
122542025-04-30T06:36:00.510ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "1bb18a9b-19e8-4bc5-b9d1-8c1b26fe1964", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
122552025-04-30T06:36:00.513ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122562025-04-30T06:36:00.513ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122572025-04-30T06:36:00.513ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122582025-04-30T06:36:00.513ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122592025-04-30T06:36:00.514ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122602025-04-30T06:36:00.514ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122612025-04-30T06:36:00.514ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122622025-04-30T06:36:00.516ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122632025-04-30T06:36:00.517ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122642025-04-30T06:36:00.517ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122652025-04-30T06:36:00.517ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
122662025-04-30T06:36:00.517ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122672025-04-30T06:36:00.521ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122682025-04-30T06:36:00.522ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
122692025-04-30T06:36:00.522ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122702025-04-30T06:36:00.522ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122712025-04-30T06:36:00.522ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122722025-04-30T06:36:00.523ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122732025-04-30T06:36:00.529ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122742025-04-30T06:36:00.535ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
122752025-04-30T06:36:00.535ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
122762025-04-30T06:36:00.535ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
122772025-04-30T06:36:00.536ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
122782025-04-30T06:36:00.552ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
122792025-04-30T06:36:00.552ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
122802025-04-30T06:36:00.552ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122812025-04-30T06:36:00.552ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122822025-04-30T06:36:00.553ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122832025-04-30T06:36:00.662ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
122842025-04-30T06:36:00.662ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
122852025-04-30T06:36:00.663ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:57060 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
122862025-04-30T06:36:00.664ZDEBGtest_metrics (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122872025-04-30T06:36:00.664ZDEBGtest_metrics (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
122882025-04-30T06:36:00.665ZDEBGtest_metrics (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
122892025-04-30T06:36:00.666ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
122902025-04-30T06:36:00.667ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
122912025-04-30T06:36:00.677ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
122922025-04-30T06:36:00.677ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
122932025-04-30T06:36:00.677ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
122942025-04-30T06:36:00.678ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
122952025-04-30T06:36:00.680ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:57060 req_id = 23399799-2fad-4709-8cb4-d10d0020b37b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122962025-04-30T06:36:00.683ZINFOtest_metrics (dropshot): request completed latency_us = 3254 local_addr = [::1]:56237 method = GET remote_addr = [::1]:57060 req_id = 23399799-2fad-4709-8cb4-d10d0020b37b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
122972025-04-30T06:36:00.685ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
122982025-04-30T06:36:00.685ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
122992025-04-30T06:36:00.686ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "23399799-2fad-4709-8cb4-d10d0020b37b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:00 GMT"} })
123002025-04-30T06:36:00.688ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
123012025-04-30T06:36:00.689ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
123022025-04-30T06:36:00.689ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
123032025-04-30T06:36:00.693ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
123042025-04-30T06:36:00.700ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123052025-04-30T06:36:00.708ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123062025-04-30T06:36:00.708ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123072025-04-30T06:36:00.709ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123082025-04-30T06:36:00.710ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123092025-04-30T06:36:00.710ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123102025-04-30T06:36:00.710ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123112025-04-30T06:36:00.716ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = measurements_i64
123122025-04-30T06:36:00.716ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
123132025-04-30T06:36:00.716ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123142025-04-30T06:36:00.721ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123152025-04-30T06:36:00.721ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123162025-04-30T06:36:00.721ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123172025-04-30T06:36:00.722ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123182025-04-30T06:36:00.778ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
123192025-04-30T06:36:00.889ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
123202025-04-30T06:36:00.890ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
123212025-04-30T06:36:00.890ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.096772206s iteration = 43 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
123222025-04-30T06:36:00.890ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 44 reason = Signaled
123232025-04-30T06:36:00.890ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
123242025-04-30T06:36:00.890ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123252025-04-30T06:36:00.890ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123262025-04-30T06:36:00.890ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123272025-04-30T06:36:00.890ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123282025-04-30T06:36:00.893ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123292025-04-30T06:36:00.900ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
123302025-04-30T06:36:00.900ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
123312025-04-30T06:36:00.900ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
123322025-04-30T06:36:00.901ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
123332025-04-30T06:36:00.911ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
123342025-04-30T06:36:00.927ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
123352025-04-30T06:36:00.927ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
123362025-04-30T06:36:00.929ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 10 1d 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61406 req_id = 1a25eeef-6e38-4719-93a7-f86b9ec8368b
123372025-04-30T06:36:00.929ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4125,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61406 req_id = 1a25eeef-6e38-4719-93a7-f86b9ec8368b
123382025-04-30T06:36:00.929ZDEBGtest_metrics (store): query key key = _mgs._tcp
123392025-04-30T06:36:00.930ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123402025-04-30T06:36:00.930ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61406 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1a25eeef-6e38-4719-93a7-f86b9ec8368b
123412025-04-30T06:36:00.930ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3825, tv_nsec: 729734884 } })
123422025-04-30T06:36:00.930ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
123432025-04-30T06:36:00.931ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 2b f0 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e +...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55719 req_id = fe38d2fc-9a42-4557-968b-0f83c62e1901
123442025-04-30T06:36:00.931ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11248,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55719 req_id = fe38d2fc-9a42-4557-968b-0f83c62e1901
123452025-04-30T06:36:00.931ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
123462025-04-30T06:36:00.931ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55719 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = fe38d2fc-9a42-4557-968b-0f83c62e1901
123472025-04-30T06:36:01.151ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
123482025-04-30T06:36:01.152ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
123492025-04-30T06:36:01.152ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
123502025-04-30T06:36:01.152ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123512025-04-30T06:36:01.152ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123522025-04-30T06:36:01.154ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123532025-04-30T06:36:01.156ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: b2 54 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c .T..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:57858 req_id = 744693c2-5dd7-422d-b424-3c1997d6fc23
123542025-04-30T06:36:01.157ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45652,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57858 req_id = 744693c2-5dd7-422d-b424-3c1997d6fc23
123552025-04-30T06:36:01.158ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
123562025-04-30T06:36:01.158ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 45652, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:57858 req_id = 744693c2-5dd7-422d-b424-3c1997d6fc23
123572025-04-30T06:36:01.163ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
123582025-04-30T06:36:01.212ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
123592025-04-30T06:36:01.212ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
123602025-04-30T06:36:01.212ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
123612025-04-30T06:36:01.219ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
123622025-04-30T06:36:01.219ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123632025-04-30T06:36:01.220ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123642025-04-30T06:36:01.223ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123652025-04-30T06:36:01.227ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "7113a47a-69d1-4ef6-b024-87898860021a", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123662025-04-30T06:36:01.248ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
123672025-04-30T06:36:01.248ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
123682025-04-30T06:36:01.249ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
123692025-04-30T06:36:01.253ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
123702025-04-30T06:36:01.254ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "fec520dc-faf3-4131-a9ef-1ab647cd53c8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123712025-04-30T06:36:01.254ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
123722025-04-30T06:36:01.258ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1099679-ebde-48fd-9053-5181184eca8b", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123732025-04-30T06:36:01.258ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
123742025-04-30T06:36:01.293ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "364b56a1-f224-4301-8e4c-75a89d1f9a1c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123752025-04-30T06:36:01.316ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
123762025-04-30T06:36:01.317ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
123772025-04-30T06:36:01.317ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123782025-04-30T06:36:01.318ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 6fbe6424-9dde-4a92-980c-785b3aaf0649 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123792025-04-30T06:36:01.319ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 6fbe6424-9dde-4a92-980c-785b3aaf0649 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123802025-04-30T06:36:01.319ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 6fbe6424-9dde-4a92-980c-785b3aaf0649 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123812025-04-30T06:36:01.337ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 6fbe6424-9dde-4a92-980c-785b3aaf0649 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123822025-04-30T06:36:01.384ZINFOtest_metrics (dropshot_internal): request completed latency_us = 65783 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 6fbe6424-9dde-4a92-980c-785b3aaf0649 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
123832025-04-30T06:36:01.385ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fbe6424-9dde-4a92-980c-785b3aaf0649", "content-length": "405", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123842025-04-30T06:36:01.386ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
123852025-04-30T06:36:01.387ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f8d48c37-8973-4ae6-8359-efeb90de2549 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
123862025-04-30T06:36:01.389ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f8d48c37-8973-4ae6-8359-efeb90de2549 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
123872025-04-30T06:36:01.389ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f8d48c37-8973-4ae6-8359-efeb90de2549 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
123882025-04-30T06:36:01.390ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f8d48c37-8973-4ae6-8359-efeb90de2549 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
123892025-04-30T06:36:01.410ZINFOtest_metrics (dropshot_internal): request completed latency_us = 23292 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f8d48c37-8973-4ae6-8359-efeb90de2549 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
123902025-04-30T06:36:01.411ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f8d48c37-8973-4ae6-8359-efeb90de2549", "content-length": "29", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123912025-04-30T06:36:01.411ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
123922025-04-30T06:36:01.411ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
123932025-04-30T06:36:01.411ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
123942025-04-30T06:36:01.434ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
123952025-04-30T06:36:01.465ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c48d0c0e-add8-4d2d-9763-60144452dcd0", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123962025-04-30T06:36:01.465ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
123972025-04-30T06:36:01.475ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "54cb9bfd-78b4-4845-995a-4a1124c14307", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
123982025-04-30T06:36:01.476ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
123992025-04-30T06:36:01.479ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "71d3f428-24e2-446c-9095-5a27b8fb64ed", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124002025-04-30T06:36:01.479ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
124012025-04-30T06:36:01.482ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e6905c42-5dd4-4de0-960a-2e1f9deff7d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124022025-04-30T06:36:01.482ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
124032025-04-30T06:36:01.485ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124042025-04-30T06:36:01.485ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124052025-04-30T06:36:01.485ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124062025-04-30T06:36:01.485ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124072025-04-30T06:36:01.486ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124082025-04-30T06:36:01.486ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e64cc0c5-479e-40bf-865e-2aa43e12b4f7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124092025-04-30T06:36:01.486ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
124102025-04-30T06:36:01.501ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "44f52396-fdcb-426d-b8cb-9a0c08393f14", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124112025-04-30T06:36:01.501ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
124122025-04-30T06:36:01.502ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124132025-04-30T06:36:01.508ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124142025-04-30T06:36:01.508ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124152025-04-30T06:36:01.508ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124162025-04-30T06:36:01.508ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124172025-04-30T06:36:01.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a52a71f3-dfc8-4dac-92e6-8b5e5ce49844", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124182025-04-30T06:36:01.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
124192025-04-30T06:36:01.517ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124202025-04-30T06:36:01.518ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124212025-04-30T06:36:01.520ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124222025-04-30T06:36:01.520ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124232025-04-30T06:36:01.522ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124242025-04-30T06:36:01.522ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124252025-04-30T06:36:01.523ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124262025-04-30T06:36:01.523ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124272025-04-30T06:36:01.523ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
124282025-04-30T06:36:01.524ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9d75b25-9289-4619-a6a1-c4125ece57f9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124292025-04-30T06:36:01.524ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
124302025-04-30T06:36:01.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec79fc2e-6554-43ad-ab33-0f90ce108d71", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124312025-04-30T06:36:01.527ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
124322025-04-30T06:36:01.531ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb7c6531-7c21-48d0-98b6-9952777c5efc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124332025-04-30T06:36:01.532ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
124342025-04-30T06:36:01.544ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "69d4af70-98cb-43e7-9a07-aace2d787642", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124352025-04-30T06:36:01.544ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
124362025-04-30T06:36:01.555ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "790e9312-4419-4b06-bd11-7354b4c2495c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124372025-04-30T06:36:01.555ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
124382025-04-30T06:36:01.565ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2868fc17-0a9d-4826-b2e1-3196091b2f1c", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124392025-04-30T06:36:01.565ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
124402025-04-30T06:36:01.566ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
124412025-04-30T06:36:01.572ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124422025-04-30T06:36:01.572ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124432025-04-30T06:36:01.573ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
124442025-04-30T06:36:01.574ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
124452025-04-30T06:36:01.600ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
124462025-04-30T06:36:01.601ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
124472025-04-30T06:36:01.604ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
124482025-04-30T06:36:01.604ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
124492025-04-30T06:36:01.605ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "743e6af5-28ea-44b9-868c-9e30276cd5a8", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124502025-04-30T06:36:01.606ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
124512025-04-30T06:36:01.608ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4ccf579-efc1-4f0a-9457-25f0b73f2e8f", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124522025-04-30T06:36:01.608ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
124532025-04-30T06:36:01.609ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ccab3f7b-4ee5-4fa1-a3e4-0c980287571f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124542025-04-30T06:36:01.609ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
124552025-04-30T06:36:01.611ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9e931a0-7e3f-4b59-b2cb-6aa6bb6caa94", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124562025-04-30T06:36:01.611ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
124572025-04-30T06:36:01.612ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6fc855ca-227a-4666-a04b-7ee4ed02b77e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124582025-04-30T06:36:01.612ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
124592025-04-30T06:36:01.615ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb8fff26-c418-46d5-97bf-203d0c11ee67", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124602025-04-30T06:36:01.615ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
124612025-04-30T06:36:01.630ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
124622025-04-30T06:36:01.630ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
124632025-04-30T06:36:01.630ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124642025-04-30T06:36:01.630ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124652025-04-30T06:36:01.631ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124662025-04-30T06:36:01.634ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6b8b530-d6dc-48dc-8f40-7e296fc439cf", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124672025-04-30T06:36:01.634ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
124682025-04-30T06:36:01.634ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
124692025-04-30T06:36:01.653ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
124702025-04-30T06:36:01.653ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124712025-04-30T06:36:01.653ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124722025-04-30T06:36:01.654ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124732025-04-30T06:36:01.660ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "26926be9-c0fb-4b23-88eb-b81651cfeab6", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124742025-04-30T06:36:01.660ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
124752025-04-30T06:36:01.661ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
124762025-04-30T06:36:01.661ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
124772025-04-30T06:36:01.662ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
124782025-04-30T06:36:01.664ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "97b93289-52e5-420a-a1b9-e35c23687fae", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124792025-04-30T06:36:01.665ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
124802025-04-30T06:36:01.701ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
124812025-04-30T06:36:01.718ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
124822025-04-30T06:36:01.722ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "92af7f7b-e290-48ff-9894-c2ed7dd34792", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
124832025-04-30T06:36:01.725ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
124842025-04-30T06:36:01.728ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ca b4 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:63520 req_id = ef699e3a-264e-4c49-b360-8a7b52272659
124852025-04-30T06:36:01.739ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51892,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63520 req_id = ef699e3a-264e-4c49-b360-8a7b52272659
124862025-04-30T06:36:01.754ZDEBGtest_metrics (store): query key key = _dendrite._tcp
124872025-04-30T06:36:01.755ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124882025-04-30T06:36:01.755ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63520 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ef699e3a-264e-4c49-b360-8a7b52272659
124892025-04-30T06:36:01.756ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3826, tv_nsec: 555042627 } })
124902025-04-30T06:36:01.756ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
124912025-04-30T06:36:01.758ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: eb 38 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .8..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60195 req_id = e94468dd-566b-44bf-9af9-62f0a84245d6
124922025-04-30T06:36:01.758ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 60216,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60195 req_id = e94468dd-566b-44bf-9af9-62f0a84245d6
124932025-04-30T06:36:01.759ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
124942025-04-30T06:36:01.759ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60195 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e94468dd-566b-44bf-9af9-62f0a84245d6
124952025-04-30T06:36:01.760ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
124962025-04-30T06:36:01.828ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
124972025-04-30T06:36:01.828ZDEBGtest_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
124982025-04-30T06:36:01.828ZDEBGtest_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
124992025-04-30T06:36:01.837ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
125002025-04-30T06:36:01.842ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
125012025-04-30T06:36:01.842ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125022025-04-30T06:36:01.843ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: ba 6c 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .l..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:53541 req_id = 64cac908-4897-4e98-8fe9-4a0030d3390a
125032025-04-30T06:36:01.844ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 47724,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53541 req_id = 64cac908-4897-4e98-8fe9-4a0030d3390a
125042025-04-30T06:36:01.844ZDEBGtest_metrics (store): query key key = _dendrite._tcp
125052025-04-30T06:36:01.844ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125062025-04-30T06:36:01.845ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:53541 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 64cac908-4897-4e98-8fe9-4a0030d3390a
125072025-04-30T06:36:01.846ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3826, tv_nsec: 645625577 } })
125082025-04-30T06:36:01.847ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
125092025-04-30T06:36:01.847ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 1a 7d 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .}..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:59618 req_id = 75cdd262-4012-43ad-913d-4537ecb3f66d
125102025-04-30T06:36:01.847ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 6781,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59618 req_id = 75cdd262-4012-43ad-913d-4537ecb3f66d
125112025-04-30T06:36:01.847ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125122025-04-30T06:36:01.847ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:59618 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 75cdd262-4012-43ad-913d-4537ecb3f66d
125132025-04-30T06:36:01.848ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
125142025-04-30T06:36:01.927ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
125152025-04-30T06:36:01.928ZDEBGtest_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
125162025-04-30T06:36:01.928ZDEBGtest_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
125172025-04-30T06:36:01.928ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
125182025-04-30T06:36:01.932ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
125192025-04-30T06:36:01.933ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "17aae0aa-583e-485d-8175-d0ec003a79c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125202025-04-30T06:36:01.933ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
125212025-04-30T06:36:01.936ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "907bcbf2-c060-44a4-8fb1-a06a9004e7cb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125222025-04-30T06:36:01.937ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
125232025-04-30T06:36:01.942ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
125242025-04-30T06:36:01.955ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "61c41586-bb20-4a53-ae18-cbaed4f0e75e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125252025-04-30T06:36:01.955ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
125262025-04-30T06:36:01.956ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "770ce38c-366d-4c32-a5d4-924960b92ca0", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125272025-04-30T06:36:01.957ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
125282025-04-30T06:36:01.958ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dc13b6f-795b-4c5d-9a35-9dea0fc6d37c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125292025-04-30T06:36:01.958ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
125302025-04-30T06:36:01.960ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5f17848-5dcd-45a2-93c6-f3322795b805", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125312025-04-30T06:36:01.960ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
125322025-04-30T06:36:01.961ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "797b8e84-d71c-4049-9a86-3a9d9ab4e419", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125332025-04-30T06:36:01.961ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
125342025-04-30T06:36:01.962ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "908be456-5c02-42a4-a553-ab749473f2d5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125352025-04-30T06:36:01.962ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
125362025-04-30T06:36:01.964ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4529d4cf-bded-432b-a32f-3fe5b389929b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125372025-04-30T06:36:01.964ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
125382025-04-30T06:36:01.967ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9ebac36-01b2-47dd-bafd-6cd9ffc42b42", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125392025-04-30T06:36:01.967ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
125402025-04-30T06:36:01.967ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
125412025-04-30T06:36:01.968ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 3d 24 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 =$..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58611 req_id = e09ed840-fdc7-4ce0-b8b5-59ec16ff482e
125422025-04-30T06:36:01.968ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15652,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58611 req_id = e09ed840-fdc7-4ce0-b8b5-59ec16ff482e
125432025-04-30T06:36:01.968ZDEBGtest_metrics (store): query key key = _dendrite._tcp
125442025-04-30T06:36:01.968ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125452025-04-30T06:36:01.969ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58611 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = e09ed840-fdc7-4ce0-b8b5-59ec16ff482e
125462025-04-30T06:36:01.969ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3826, tv_nsec: 768241341 } })
125472025-04-30T06:36:01.969ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
125482025-04-30T06:36:01.969ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: af 26 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .&..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:62220 req_id = 53d2b807-64b9-44ef-be8f-6e4a36ae1c93
125492025-04-30T06:36:01.970ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 44838,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62220 req_id = 53d2b807-64b9-44ef-be8f-6e4a36ae1c93
125502025-04-30T06:36:01.970ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
125512025-04-30T06:36:01.970ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:62220 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 53d2b807-64b9-44ef-be8f-6e4a36ae1c93
125522025-04-30T06:36:01.970ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
125532025-04-30T06:36:02.052ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
125542025-04-30T06:36:02.053ZDEBGtest_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
125552025-04-30T06:36:02.053ZDEBGtest_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
125562025-04-30T06:36:02.053ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
125572025-04-30T06:36:02.059ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
125582025-04-30T06:36:02.060ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
125592025-04-30T06:36:02.060ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125602025-04-30T06:36:02.060ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125612025-04-30T06:36:02.060ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125622025-04-30T06:36:02.061ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125632025-04-30T06:36:02.063ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "cff4f9b7-73aa-4b44-8322-9c1dfc35306a", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125642025-04-30T06:36:02.064ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
125652025-04-30T06:36:02.064ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125662025-04-30T06:36:02.070ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
125672025-04-30T06:36:02.070ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125682025-04-30T06:36:02.070ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125692025-04-30T06:36:02.070ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125702025-04-30T06:36:02.073ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
125712025-04-30T06:36:02.073ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
125722025-04-30T06:36:02.074ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
125732025-04-30T06:36:02.081ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d10084e3-d5f5-443b-9191-549a076d7616", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125742025-04-30T06:36:02.081ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
125752025-04-30T06:36:02.092ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c2057fcd-76de-4eb6-bd07-c6819ce5fc40", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125762025-04-30T06:36:02.092ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
125772025-04-30T06:36:02.098ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "e90addf6-9cc0-460c-ae5f-61b128132f01", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125782025-04-30T06:36:02.098ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
125792025-04-30T06:36:02.103ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "603a9700-3a1f-4213-9d1f-72916d940f80", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125802025-04-30T06:36:02.104ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
125812025-04-30T06:36:02.106ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c633992b-f312-4edf-8e3f-9c7dde76e4de", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125822025-04-30T06:36:02.107ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
125832025-04-30T06:36:02.108ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8db4cd43-b694-4c4b-94a4-595d79dd9eca", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125842025-04-30T06:36:02.109ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
125852025-04-30T06:36:02.114ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f153ad26-c3a0-402a-ac51-0a1b94ecc783", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:01 GMT"} })
125862025-04-30T06:36:02.114ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
125872025-04-30T06:36:02.115ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125882025-04-30T06:36:02.115ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125892025-04-30T06:36:02.116ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125902025-04-30T06:36:02.116ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125912025-04-30T06:36:02.118ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
125922025-04-30T06:36:02.119ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
125932025-04-30T06:36:02.132ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
125942025-04-30T06:36:02.132ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
125952025-04-30T06:36:02.132ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
125962025-04-30T06:36:02.132ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
125972025-04-30T06:36:02.132ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
125982025-04-30T06:36:02.133ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
125992025-04-30T06:36:02.136ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126002025-04-30T06:36:02.145ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
126012025-04-30T06:36:02.145ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126022025-04-30T06:36:02.145ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126032025-04-30T06:36:02.146ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126042025-04-30T06:36:02.293ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
126052025-04-30T06:36:02.293ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
126062025-04-30T06:36:02.295ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126072025-04-30T06:36:02.296ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126082025-04-30T06:36:02.296ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126092025-04-30T06:36:02.296ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126102025-04-30T06:36:02.298ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126112025-04-30T06:36:02.299ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:64828 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
126122025-04-30T06:36:02.300ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:64828 req_id = 6b8bad5c-2a89-4af4-bd01-96d183a30128 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126132025-04-30T06:36:02.304ZINFOtest_metrics (dropshot): request completed latency_us = 4474 local_addr = [::1]:56237 method = GET remote_addr = [::1]:64828 req_id = 6b8bad5c-2a89-4af4-bd01-96d183a30128 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
126142025-04-30T06:36:02.305ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6b8bad5c-2a89-4af4-bd01-96d183a30128", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
126152025-04-30T06:36:02.311ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126162025-04-30T06:36:02.311ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
126172025-04-30T06:36:02.311ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
126182025-04-30T06:36:02.313ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126192025-04-30T06:36:02.316ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126202025-04-30T06:36:02.319ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126212025-04-30T06:36:02.320ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126222025-04-30T06:36:02.320ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126232025-04-30T06:36:02.321ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126242025-04-30T06:36:02.346ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126252025-04-30T06:36:02.346ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126262025-04-30T06:36:02.346ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126272025-04-30T06:36:02.347ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126282025-04-30T06:36:02.351ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126292025-04-30T06:36:02.376ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126302025-04-30T06:36:02.377ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126312025-04-30T06:36:02.377ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126322025-04-30T06:36:02.377ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126332025-04-30T06:36:02.394ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126342025-04-30T06:36:02.394ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126352025-04-30T06:36:02.394ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126362025-04-30T06:36:02.486ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
126372025-04-30T06:36:02.486ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
126382025-04-30T06:36:02.487ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.596853096s iteration = 44 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
126392025-04-30T06:36:02.487ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 45 reason = Signaled
126402025-04-30T06:36:02.487ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
126412025-04-30T06:36:02.487ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126422025-04-30T06:36:02.487ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126432025-04-30T06:36:02.487ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126442025-04-30T06:36:02.488ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126452025-04-30T06:36:02.491ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126462025-04-30T06:36:02.495ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
126472025-04-30T06:36:02.495ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
126482025-04-30T06:36:02.495ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
126492025-04-30T06:36:02.496ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
126502025-04-30T06:36:02.514ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126512025-04-30T06:36:02.514ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126522025-04-30T06:36:02.515ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
126532025-04-30T06:36:02.515ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
126542025-04-30T06:36:02.518ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
126552025-04-30T06:36:02.522ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 6e 67 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ng..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:49192 req_id = 410eac39-5109-40c0-adda-1035eefd085f
126562025-04-30T06:36:02.522ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28263,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49192 req_id = 410eac39-5109-40c0-adda-1035eefd085f
126572025-04-30T06:36:02.522ZDEBGtest_metrics (store): query key key = _mgs._tcp
126582025-04-30T06:36:02.523ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126592025-04-30T06:36:02.523ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49192 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 410eac39-5109-40c0-adda-1035eefd085f
126602025-04-30T06:36:02.523ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126612025-04-30T06:36:02.524ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126622025-04-30T06:36:02.524ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126632025-04-30T06:36:02.524ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
126642025-04-30T06:36:02.525ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3827, tv_nsec: 324229681 } })
126652025-04-30T06:36:02.525ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
126662025-04-30T06:36:02.525ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 65 52 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e eR..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53271 req_id = 6db7cebb-2390-413a-8ac2-2ce785505c81
126672025-04-30T06:36:02.526ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 25938,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53271 req_id = 6db7cebb-2390-413a-8ac2-2ce785505c81
126682025-04-30T06:36:02.526ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
126692025-04-30T06:36:02.526ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53271 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 6db7cebb-2390-413a-8ac2-2ce785505c81
126702025-04-30T06:36:02.703ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
126712025-04-30T06:36:02.703ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
126722025-04-30T06:36:02.703ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
126732025-04-30T06:36:02.703ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126742025-04-30T06:36:02.703ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126752025-04-30T06:36:02.704ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126762025-04-30T06:36:02.704ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
126772025-04-30T06:36:02.705ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
126782025-04-30T06:36:02.709ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
126792025-04-30T06:36:02.711ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
126802025-04-30T06:36:02.720ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 11 83 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49847 req_id = 50f4ebec-131a-46d7-aebc-f39ed5b62f75
126812025-04-30T06:36:02.720ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4483,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49847 req_id = 50f4ebec-131a-46d7-aebc-f39ed5b62f75
126822025-04-30T06:36:02.721ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
126832025-04-30T06:36:02.721ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 4483, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49847 req_id = 50f4ebec-131a-46d7-aebc-f39ed5b62f75
126842025-04-30T06:36:02.722ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
126852025-04-30T06:36:02.728ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
126862025-04-30T06:36:02.733ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126872025-04-30T06:36:02.733ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126882025-04-30T06:36:02.754ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126892025-04-30T06:36:02.754ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
126902025-04-30T06:36:02.754ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
126912025-04-30T06:36:02.754ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
126922025-04-30T06:36:02.754ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
126932025-04-30T06:36:02.754ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
126942025-04-30T06:36:02.807ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
126952025-04-30T06:36:02.807ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
126962025-04-30T06:36:02.810ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
126972025-04-30T06:36:02.813ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
126982025-04-30T06:36:02.814ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
126992025-04-30T06:36:02.814ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
127002025-04-30T06:36:02.817ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "d9c789f1-4a73-403c-bc0e-3fb16cbaea29", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127012025-04-30T06:36:02.818ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
127022025-04-30T06:36:02.822ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a79517d7-6bd9-41d4-90bc-136986a6e0d8", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127032025-04-30T06:36:02.823ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
127042025-04-30T06:36:02.827ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9536b67b-f495-4a54-a21d-38caee876c66", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127052025-04-30T06:36:02.827ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
127062025-04-30T06:36:02.830ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9eb42b1-58a6-4f3d-a688-6d37987a1273", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127072025-04-30T06:36:02.830ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
127082025-04-30T06:36:02.834ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b14c15c-c834-4115-b21e-13a464d45b47", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127092025-04-30T06:36:02.834ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
127102025-04-30T06:36:02.837ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5de44b8d-1206-487b-a75c-901417c276c3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127112025-04-30T06:36:02.849ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
127122025-04-30T06:36:02.856ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ef5538b-5a5d-4bbc-92f9-a2999fc2392b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127132025-04-30T06:36:02.857ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
127142025-04-30T06:36:02.864ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
127152025-04-30T06:36:02.867ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "72864ecb-6df8-4a55-b032-79d7d71a7890", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127162025-04-30T06:36:02.867ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
127172025-04-30T06:36:02.873ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d6947bb-c56e-468b-9f7f-b319f6e52946", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127182025-04-30T06:36:02.873ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
127192025-04-30T06:36:02.874ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e57c3cde-62dc-43bd-9cfe-ce5d854af454", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127202025-04-30T06:36:02.905ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
127212025-04-30T06:36:02.908ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5127b9d6-5cfc-410c-8118-019200e18c04", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127222025-04-30T06:36:02.908ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
127232025-04-30T06:36:02.916ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe31e7f5-65db-43b4-b90a-864c8be4ff33", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127242025-04-30T06:36:02.919ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
127252025-04-30T06:36:02.920ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127262025-04-30T06:36:02.920ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127272025-04-30T06:36:02.920ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127282025-04-30T06:36:02.920ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127292025-04-30T06:36:02.921ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127302025-04-30T06:36:02.930ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "af400b9d-8ff6-4085-bf7c-d434f390202b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127312025-04-30T06:36:02.930ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
127322025-04-30T06:36:02.930ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127332025-04-30T06:36:02.937ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127342025-04-30T06:36:02.938ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127352025-04-30T06:36:02.938ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127362025-04-30T06:36:02.939ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127372025-04-30T06:36:02.947ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127382025-04-30T06:36:02.947ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127392025-04-30T06:36:02.948ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127402025-04-30T06:36:02.949ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ac34c14-8c4e-4345-a39f-4b1fac10ae2c", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127412025-04-30T06:36:02.949ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
127422025-04-30T06:36:02.990ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "25aad379-c9ed-4c8a-9afc-a98664323d3f", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127432025-04-30T06:36:02.990ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
127442025-04-30T06:36:02.994ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee6a3830-0dcb-42db-8b43-0242bf6d8c95", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127452025-04-30T06:36:02.994ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
127462025-04-30T06:36:03.015ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "81e33791-6f3b-410c-a0ce-677899c35626", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127472025-04-30T06:36:03.015ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
127482025-04-30T06:36:03.017ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "805f3122-e671-44f9-99c7-9194980e8666", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127492025-04-30T06:36:03.017ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
127502025-04-30T06:36:03.019ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e91a8ee-6161-4da5-aba7-55d73b73ba03", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127512025-04-30T06:36:03.019ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
127522025-04-30T06:36:03.019ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "174fcd68-6b7c-4f56-a3d9-387cd5fe364e", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127532025-04-30T06:36:03.020ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
127542025-04-30T06:36:03.021ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf417376-3b52-46f0-91fc-a0cc3ea4965a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127552025-04-30T06:36:03.021ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
127562025-04-30T06:36:03.023ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d44a286c-6992-4979-a063-0ab90eb7a83c", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127572025-04-30T06:36:03.023ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
127582025-04-30T06:36:03.026ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ab19ad4-2603-4ba2-8f68-91153e1a8ceb", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127592025-04-30T06:36:03.027ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
127602025-04-30T06:36:03.030ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3cb5211d-e1f2-417b-9281-8589dcf043a9", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127612025-04-30T06:36:03.030ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
127622025-04-30T06:36:03.036ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
127632025-04-30T06:36:03.037ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4bcbe3ca-b372-44d5-bd88-548e81a5e40e", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127642025-04-30T06:36:03.037ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
127652025-04-30T06:36:03.051ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d456b6c-5514-4963-aa4f-38c92394e459", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127662025-04-30T06:36:03.052ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
127672025-04-30T06:36:03.071ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "aee3133b-a3bb-4bbf-bd48-4ac6afc7b591", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127682025-04-30T06:36:03.071ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
127692025-04-30T06:36:03.072ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6d965ff-caae-46f0-8015-773334b28031", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127702025-04-30T06:36:03.072ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
127712025-04-30T06:36:03.074ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b12a913-f1d4-4548-a7b9-e190a1551d0d", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127722025-04-30T06:36:03.074ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
127732025-04-30T06:36:03.076ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "5efe5a0a-84ed-4681-b4a5-e096990a03ab", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127742025-04-30T06:36:03.076ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
127752025-04-30T06:36:03.077ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ece54cf1-e431-41c6-8767-065edb088042", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127762025-04-30T06:36:03.077ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
127772025-04-30T06:36:03.078ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d4083873-0d4f-4651-96ce-646267c97575", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127782025-04-30T06:36:03.078ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
127792025-04-30T06:36:03.079ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e0f87c5-bee4-4547-aff2-4d40e15787fc", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127802025-04-30T06:36:03.079ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
127812025-04-30T06:36:03.080ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5dfb0904-ccf5-4c8a-a5b9-60a1d8d64eb0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127822025-04-30T06:36:03.080ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
127832025-04-30T06:36:03.081ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "31c9362b-5254-47fa-ae11-e79bb170eba1", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127842025-04-30T06:36:03.082ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
127852025-04-30T06:36:03.084ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4e908e72-2e10-45db-b56b-774f086e2842", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127862025-04-30T06:36:03.084ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
127872025-04-30T06:36:03.086ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ebb3bb4-fb0d-48a8-8e33-740fed421378", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
127882025-04-30T06:36:03.087ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
127892025-04-30T06:36:03.087ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
127902025-04-30T06:36:03.087ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
127912025-04-30T06:36:03.087ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127922025-04-30T06:36:03.087ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127932025-04-30T06:36:03.088ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127942025-04-30T06:36:03.090ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
127952025-04-30T06:36:03.095ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
127962025-04-30T06:36:03.095ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
127972025-04-30T06:36:03.095ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
127982025-04-30T06:36:03.095ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
127992025-04-30T06:36:03.098ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad0cdb17-1d45-4357-ab6e-3ace27a37433", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128002025-04-30T06:36:03.098ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
128012025-04-30T06:36:03.098ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128022025-04-30T06:36:03.098ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128032025-04-30T06:36:03.099ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128042025-04-30T06:36:03.107ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "48390954-7ae1-47d8-a81c-1bb95ac90f69", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128052025-04-30T06:36:03.107ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
128062025-04-30T06:36:03.109ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "68cb2938-3c84-43b0-9272-d7addc3bb33f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128072025-04-30T06:36:03.113ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
128082025-04-30T06:36:03.119ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "500658b4-3d71-4387-bc32-48f39a628a5f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128092025-04-30T06:36:03.119ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
128102025-04-30T06:36:03.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2912e65f-546f-4b85-b388-2fd3f1f63dbd", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128112025-04-30T06:36:03.121ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
128122025-04-30T06:36:03.121ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9aab8c4e-b1dd-48f1-9459-deec76ed40d4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128132025-04-30T06:36:03.121ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
128142025-04-30T06:36:03.124ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ea1bcff-ccf6-4bcb-955b-02835fa3b017", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128152025-04-30T06:36:03.125ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
128162025-04-30T06:36:03.127ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "07a72943-b9d7-46df-8b0f-66e81e2578f5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128172025-04-30T06:36:03.127ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128182025-04-30T06:36:03.127ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128192025-04-30T06:36:03.127ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128202025-04-30T06:36:03.127ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128212025-04-30T06:36:03.127ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128222025-04-30T06:36:03.132ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128232025-04-30T06:36:03.136ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128242025-04-30T06:36:03.136ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
128252025-04-30T06:36:03.136ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128262025-04-30T06:36:03.136ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128272025-04-30T06:36:03.136ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128282025-04-30T06:36:03.139ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128292025-04-30T06:36:03.149ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128302025-04-30T06:36:03.163ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
128312025-04-30T06:36:03.163ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128322025-04-30T06:36:03.163ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128332025-04-30T06:36:03.163ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128342025-04-30T06:36:03.252ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
128352025-04-30T06:36:03.253ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
128362025-04-30T06:36:03.254ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
128372025-04-30T06:36:03.255ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:55294 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
128382025-04-30T06:36:03.255ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:55294 req_id = 9142c443-d87c-4092-888e-d352cf5bd34b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128392025-04-30T06:36:03.256ZINFOtest_metrics (dropshot): request completed latency_us = 513 local_addr = [::1]:56237 method = GET remote_addr = [::1]:55294 req_id = 9142c443-d87c-4092-888e-d352cf5bd34b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
128402025-04-30T06:36:03.257ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9142c443-d87c-4092-888e-d352cf5bd34b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:02 GMT"} })
128412025-04-30T06:36:03.260ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128422025-04-30T06:36:03.260ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
128432025-04-30T06:36:03.260ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
128442025-04-30T06:36:03.263ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128452025-04-30T06:36:03.263ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128462025-04-30T06:36:03.263ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128472025-04-30T06:36:03.264ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128482025-04-30T06:36:03.265ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128492025-04-30T06:36:03.285ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128502025-04-30T06:36:03.299ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128512025-04-30T06:36:03.299ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128522025-04-30T06:36:03.299ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128532025-04-30T06:36:03.300ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128542025-04-30T06:36:03.301ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
128552025-04-30T06:36:03.301ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128562025-04-30T06:36:03.302ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 2e616b55-efde-4c30-90eb-cc9b18273379 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128572025-04-30T06:36:03.302ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 2e616b55-efde-4c30-90eb-cc9b18273379 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128582025-04-30T06:36:03.302ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 2e616b55-efde-4c30-90eb-cc9b18273379 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128592025-04-30T06:36:03.303ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 2e616b55-efde-4c30-90eb-cc9b18273379 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128602025-04-30T06:36:03.306ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
128612025-04-30T06:36:03.306ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128622025-04-30T06:36:03.306ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128632025-04-30T06:36:03.306ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128642025-04-30T06:36:03.307ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128652025-04-30T06:36:03.308ZINFOtest_metrics (dropshot_internal): request completed latency_us = 6150 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 2e616b55-efde-4c30-90eb-cc9b18273379 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
128662025-04-30T06:36:03.309ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "2e616b55-efde-4c30-90eb-cc9b18273379", "content-length": "405", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
128672025-04-30T06:36:03.309ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128682025-04-30T06:36:03.310ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f0af01f2-53d1-4b75-9ae6-0475a694b479 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128692025-04-30T06:36:03.310ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f0af01f2-53d1-4b75-9ae6-0475a694b479 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128702025-04-30T06:36:03.310ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f0af01f2-53d1-4b75-9ae6-0475a694b479 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128712025-04-30T06:36:03.311ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f0af01f2-53d1-4b75-9ae6-0475a694b479 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128722025-04-30T06:36:03.333ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128732025-04-30T06:36:03.339ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
128742025-04-30T06:36:03.340ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128752025-04-30T06:36:03.340ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128762025-04-30T06:36:03.341ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128772025-04-30T06:36:03.346ZINFOtest_metrics (dropshot_internal): request completed latency_us = 36193 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = f0af01f2-53d1-4b75-9ae6-0475a694b479 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
128782025-04-30T06:36:03.347ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "f0af01f2-53d1-4b75-9ae6-0475a694b479", "content-length": "29", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
128792025-04-30T06:36:03.347ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
128802025-04-30T06:36:03.347ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
128812025-04-30T06:36:03.347ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
128822025-04-30T06:36:03.378ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
128832025-04-30T06:36:03.379ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
128842025-04-30T06:36:03.379ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 891.981003ms iteration = 45 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
128852025-04-30T06:36:03.379ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 46 reason = Signaled
128862025-04-30T06:36:03.379ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
128872025-04-30T06:36:03.379ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
128882025-04-30T06:36:03.379ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128892025-04-30T06:36:03.379ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128902025-04-30T06:36:03.380ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128912025-04-30T06:36:03.381ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
128922025-04-30T06:36:03.381ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
128932025-04-30T06:36:03.382ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
128942025-04-30T06:36:03.389ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
128952025-04-30T06:36:03.393ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
128962025-04-30T06:36:03.394ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
128972025-04-30T06:36:03.394ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
128982025-04-30T06:36:03.394ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
128992025-04-30T06:36:03.400ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
129002025-04-30T06:36:03.400ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
129012025-04-30T06:36:03.403ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 0b 86 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:57045 req_id = 24648c9b-9553-43cd-98b3-374ad968f7a2
129022025-04-30T06:36:03.403ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2950,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57045 req_id = 24648c9b-9553-43cd-98b3-374ad968f7a2
129032025-04-30T06:36:03.403ZDEBGtest_metrics (store): query key key = _mgs._tcp
129042025-04-30T06:36:03.403ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129052025-04-30T06:36:03.403ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:57045 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 24648c9b-9553-43cd-98b3-374ad968f7a2
129062025-04-30T06:36:03.404ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3828, tv_nsec: 203346905 } })
129072025-04-30T06:36:03.404ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
129082025-04-30T06:36:03.405ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 3d 26 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e =&..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57200 req_id = 76667f02-a1b8-49c5-a808-f317996b72da
129092025-04-30T06:36:03.405ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15654,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57200 req_id = 76667f02-a1b8-49c5-a808-f317996b72da
129102025-04-30T06:36:03.405ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
129112025-04-30T06:36:03.405ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57200 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 76667f02-a1b8-49c5-a808-f317996b72da
129122025-04-30T06:36:03.474ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
129132025-04-30T06:36:03.489ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 2a 33 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c *3..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:49984 req_id = 84e499fc-b0f8-4402-b90f-a838074357f0
129142025-04-30T06:36:03.489ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 10803,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49984 req_id = 84e499fc-b0f8-4402-b90f-a838074357f0
129152025-04-30T06:36:03.491ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
129162025-04-30T06:36:03.491ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 10803, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:49984 req_id = 84e499fc-b0f8-4402-b90f-a838074357f0
129172025-04-30T06:36:03.491ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
129182025-04-30T06:36:03.491ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
129192025-04-30T06:36:03.491ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
129202025-04-30T06:36:03.495ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "c62d28cf-98e6-411d-a86c-d6d3a323ce29", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129212025-04-30T06:36:03.495ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
129222025-04-30T06:36:03.496ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c73dcaf3-3179-4779-90ca-4c43ab601233", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129232025-04-30T06:36:03.497ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
129242025-04-30T06:36:03.501ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6a7c46d-94a0-4f3e-abc0-f7c2992c51a2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129252025-04-30T06:36:03.501ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
129262025-04-30T06:36:03.506ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0704f371-7b94-4948-bc98-26066eafbe2d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129272025-04-30T06:36:03.506ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
129282025-04-30T06:36:03.509ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0766f32b-f505-40ef-8c1e-83644c33a177", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129292025-04-30T06:36:03.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
129302025-04-30T06:36:03.510ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d97e7bf6-76a4-4a64-8d84-fb4d9b187627", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129312025-04-30T06:36:03.510ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
129322025-04-30T06:36:03.514ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129332025-04-30T06:36:03.514ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129342025-04-30T06:36:03.515ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
129352025-04-30T06:36:03.524ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129362025-04-30T06:36:03.526ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129372025-04-30T06:36:03.526ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129382025-04-30T06:36:03.526ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
129392025-04-30T06:36:03.543ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b5d366a9-2c6e-4b51-bfbe-ae1cd83f696c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129402025-04-30T06:36:03.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
129412025-04-30T06:36:03.569ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
129422025-04-30T06:36:03.569ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
129432025-04-30T06:36:03.569ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129442025-04-30T06:36:03.569ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
129452025-04-30T06:36:03.569ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129462025-04-30T06:36:03.569ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129472025-04-30T06:36:03.569ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129482025-04-30T06:36:03.569ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
129492025-04-30T06:36:03.570ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
129502025-04-30T06:36:03.572ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
129512025-04-30T06:36:03.580ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
129522025-04-30T06:36:03.581ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129532025-04-30T06:36:03.581ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
129542025-04-30T06:36:03.581ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129552025-04-30T06:36:03.582ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
129562025-04-30T06:36:03.582ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
129572025-04-30T06:36:03.584ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
129582025-04-30T06:36:03.584ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
129592025-04-30T06:36:03.585ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
129602025-04-30T06:36:03.585ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a7daccc1-9b69-4bd8-b1f3-d7dd31213793", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129612025-04-30T06:36:03.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
129622025-04-30T06:36:03.586ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
129632025-04-30T06:36:03.586ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
129642025-04-30T06:36:03.587ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8444095a-96e9-4ae7-a6e3-486861399ca9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129652025-04-30T06:36:03.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
129662025-04-30T06:36:03.589ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4830c7ee-a2ae-43e8-bb33-565848566f35", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129672025-04-30T06:36:03.589ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
129682025-04-30T06:36:03.591ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a03cdaa8-d3ce-4a5e-a8ad-51014900382b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129692025-04-30T06:36:03.592ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
129702025-04-30T06:36:03.594ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "730c268d-ea6c-48bc-b50d-38ce7f2c16f3", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129712025-04-30T06:36:03.594ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
129722025-04-30T06:36:03.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7609c7a4-addb-4262-bfc8-a0de41e527d3", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129732025-04-30T06:36:03.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
129742025-04-30T06:36:03.605ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e5e14922-bfd0-4ce1-abd9-a440a50420ac", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129752025-04-30T06:36:03.606ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
129762025-04-30T06:36:03.609ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "dd9798af-4e8d-41ef-8d46-b78850462983", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129772025-04-30T06:36:03.609ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
129782025-04-30T06:36:03.612ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dad3b18a-584f-4445-bdcf-60e76824b641", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129792025-04-30T06:36:03.612ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
129802025-04-30T06:36:03.618ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3951523c-d7b8-41a6-b26a-d6cfb2875f9b", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129812025-04-30T06:36:03.618ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
129822025-04-30T06:36:03.621ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c21142ef-e0f1-46f7-a839-a54b4035c56d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129832025-04-30T06:36:03.621ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
129842025-04-30T06:36:03.625ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "95a5db8a-f9af-44eb-8db6-baef28871f68", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129852025-04-30T06:36:03.625ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
129862025-04-30T06:36:03.626ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
129872025-04-30T06:36:03.626ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "318223ec-0bf3-4687-a80a-d9c3b8df00c8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129882025-04-30T06:36:03.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
129892025-04-30T06:36:03.628ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c89842d2-6b4b-4092-8f57-e60e2bea49ef", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129902025-04-30T06:36:03.628ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
129912025-04-30T06:36:03.629ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2edccf15-4fbe-403a-882b-ae09914375eb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129922025-04-30T06:36:03.629ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
129932025-04-30T06:36:03.630ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "78bca30a-ef95-409c-af38-776ab3150d99", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129942025-04-30T06:36:03.630ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
129952025-04-30T06:36:03.633ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b253fb28-1fd1-4dfd-8fa3-a3d15d17a17e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129962025-04-30T06:36:03.633ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
129972025-04-30T06:36:03.634ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "483b6c35-fcee-4beb-8453-1b4eb697e571", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
129982025-04-30T06:36:03.635ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
129992025-04-30T06:36:03.642ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "023d3cb4-fa1b-4237-8797-e270b370d6c1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130002025-04-30T06:36:03.642ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
130012025-04-30T06:36:03.644ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0a5338ee-98d3-4cc3-ac97-c2a6b83f14b0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130022025-04-30T06:36:03.644ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
130032025-04-30T06:36:03.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "40c6e010-578d-41fc-95d7-d4ee3592d268", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130042025-04-30T06:36:03.646ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
130052025-04-30T06:36:03.648ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "078995ae-9075-45ae-9418-0ee55c98d483", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130062025-04-30T06:36:03.684ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
130072025-04-30T06:36:03.685ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130082025-04-30T06:36:03.685ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130092025-04-30T06:36:03.685ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130102025-04-30T06:36:03.685ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130112025-04-30T06:36:03.687ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130122025-04-30T06:36:03.688ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
130132025-04-30T06:36:03.691ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130142025-04-30T06:36:03.715ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130152025-04-30T06:36:03.715ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130162025-04-30T06:36:03.715ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130172025-04-30T06:36:03.715ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130182025-04-30T06:36:03.716ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a9e0c22b-3f76-435b-9280-22901418ddfe", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130192025-04-30T06:36:03.717ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
130202025-04-30T06:36:03.719ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130212025-04-30T06:36:03.719ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130222025-04-30T06:36:03.719ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130232025-04-30T06:36:03.720ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a56865c-20cb-4214-9afd-c32db81e0371", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130242025-04-30T06:36:03.720ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
130252025-04-30T06:36:03.722ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d9ef4eb-78a8-4e71-b806-909ecc3ad554", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130262025-04-30T06:36:03.722ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
130272025-04-30T06:36:03.724ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f6bd9626-5690-4944-8419-f2171071cf72", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130282025-04-30T06:36:03.726ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
130292025-04-30T06:36:03.728ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "94d29f2b-aa93-4aad-adb1-c4b9f903b41f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130302025-04-30T06:36:03.728ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
130312025-04-30T06:36:03.730ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c3b3f7e-8884-4990-8321-b2d1b14df10f", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130322025-04-30T06:36:03.730ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
130332025-04-30T06:36:03.733ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c11d51d-313c-4a74-9a11-2470c0da629a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130342025-04-30T06:36:03.734ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
130352025-04-30T06:36:03.737ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a92ad8f1-d82b-4fb6-920e-a53f1c92c1b1", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130362025-04-30T06:36:03.737ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
130372025-04-30T06:36:03.741ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "533e4856-3f90-488b-b195-973b70f232cc", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130382025-04-30T06:36:03.741ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
130392025-04-30T06:36:03.745ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "50b6d6af-95b3-4d98-b5f1-39e7c31e0b40", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130402025-04-30T06:36:03.745ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
130412025-04-30T06:36:03.746ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "298e22a4-f723-4010-9bbf-d8dd57cc669e", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130422025-04-30T06:36:03.747ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
130432025-04-30T06:36:03.748ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32b1c4be-fe1a-4fe9-85cb-a0f08481be3b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130442025-04-30T06:36:03.748ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
130452025-04-30T06:36:03.751ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "16ef2f12-9b8c-4374-a108-628d4538fd1d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130462025-04-30T06:36:03.751ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
130472025-04-30T06:36:03.752ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
130482025-04-30T06:36:03.753ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4778d1b3-33b8-44f5-b703-d2cd08dd39b4", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130492025-04-30T06:36:03.753ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
130502025-04-30T06:36:03.754ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "8e6ac1e5-29c5-45f3-abd5-d6e14429b57e", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130512025-04-30T06:36:03.754ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
130522025-04-30T06:36:03.755ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "396fedef-2d39-4171-8140-487af689d68d", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130532025-04-30T06:36:03.755ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130542025-04-30T06:36:03.755ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130552025-04-30T06:36:03.755ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130562025-04-30T06:36:03.755ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130572025-04-30T06:36:03.755ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130582025-04-30T06:36:03.783ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130592025-04-30T06:36:03.787ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130602025-04-30T06:36:03.787ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
130612025-04-30T06:36:03.787ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130622025-04-30T06:36:03.788ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130632025-04-30T06:36:03.788ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130642025-04-30T06:36:03.788ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130652025-04-30T06:36:03.794ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130662025-04-30T06:36:03.798ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
130672025-04-30T06:36:03.798ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
130682025-04-30T06:36:03.798ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
130692025-04-30T06:36:03.798ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
130702025-04-30T06:36:03.814ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
130712025-04-30T06:36:03.814ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
130722025-04-30T06:36:03.814ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130732025-04-30T06:36:03.814ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130742025-04-30T06:36:03.816ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130752025-04-30T06:36:03.958ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
130762025-04-30T06:36:03.958ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
130772025-04-30T06:36:03.958ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:46297 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
130782025-04-30T06:36:03.959ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
130792025-04-30T06:36:03.959ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
130802025-04-30T06:36:03.960ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
130812025-04-30T06:36:03.966ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
130822025-04-30T06:36:03.966ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
130832025-04-30T06:36:03.966ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
130842025-04-30T06:36:03.966ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
130852025-04-30T06:36:03.966ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:46297 req_id = 6e6e7814-4eda-4f33-8bfb-ce4f0080808b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130862025-04-30T06:36:03.967ZINFOtest_metrics (dropshot): request completed latency_us = 751 local_addr = [::1]:56237 method = GET remote_addr = [::1]:46297 req_id = 6e6e7814-4eda-4f33-8bfb-ce4f0080808b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
130872025-04-30T06:36:03.968ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 11 30 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .0..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:59685 req_id = efcdc06a-d002-4328-a1a2-06c05f74c86d
130882025-04-30T06:36:03.968ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 4400,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59685 req_id = efcdc06a-d002-4328-a1a2-06c05f74c86d
130892025-04-30T06:36:03.968ZDEBGtest_metrics (store): query key key = _dendrite._tcp
130902025-04-30T06:36:03.968ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130912025-04-30T06:36:03.968ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:59685 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = efcdc06a-d002-4328-a1a2-06c05f74c86d
130922025-04-30T06:36:03.968ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: e7 96 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 ............._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64042 req_id = 96aad663-b468-4e65-815e-4e979b6f12ab
130932025-04-30T06:36:03.968ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 59286,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64042 req_id = 96aad663-b468-4e65-815e-4e979b6f12ab
130942025-04-30T06:36:03.968ZDEBGtest_metrics (store): query key key = _dendrite._tcp
130952025-04-30T06:36:03.968ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
130962025-04-30T06:36:03.969ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64042 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 96aad663-b468-4e65-815e-4e979b6f12ab
130972025-04-30T06:36:03.969ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "6e6e7814-4eda-4f33-8bfb-ce4f0080808b", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:03 GMT"} })
130982025-04-30T06:36:03.971ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
130992025-04-30T06:36:03.971ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
131002025-04-30T06:36:03.972ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
131012025-04-30T06:36:03.974ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131022025-04-30T06:36:03.979ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131032025-04-30T06:36:03.979ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131042025-04-30T06:36:03.979ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131052025-04-30T06:36:03.980ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131062025-04-30T06:36:03.984ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3828, tv_nsec: 782962751 } })
131072025-04-30T06:36:03.984ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
131082025-04-30T06:36:03.984ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3828, tv_nsec: 783629880 } })
131092025-04-30T06:36:03.984ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
131102025-04-30T06:36:03.985ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131112025-04-30T06:36:03.985ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
131122025-04-30T06:36:03.986ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131132025-04-30T06:36:03.986ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131142025-04-30T06:36:03.990ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131152025-04-30T06:36:03.991ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131162025-04-30T06:36:03.991ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131172025-04-30T06:36:03.991ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131182025-04-30T06:36:03.992ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 33 d2 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e 3...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63262 req_id = e1acf108-5d5a-4d65-b281-1d2fddf4694d
131192025-04-30T06:36:03.992ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13266,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63262 req_id = e1acf108-5d5a-4d65-b281-1d2fddf4694d
131202025-04-30T06:36:03.993ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131212025-04-30T06:36:03.993ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63262 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = e1acf108-5d5a-4d65-b281-1d2fddf4694d
131222025-04-30T06:36:03.993ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: a7 9a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63756 req_id = 4e72b0ab-a0cf-4a8d-879d-3e993b3070b0
131232025-04-30T06:36:03.993ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 42906,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63756 req_id = 4e72b0ab-a0cf-4a8d-879d-3e993b3070b0
131242025-04-30T06:36:03.993ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131252025-04-30T06:36:03.994ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63756 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 4e72b0ab-a0cf-4a8d-879d-3e993b3070b0
131262025-04-30T06:36:03.994ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
131272025-04-30T06:36:04.070ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
131282025-04-30T06:36:04.070ZDEBGtest_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
131292025-04-30T06:36:04.070ZDEBGtest_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
131302025-04-30T06:36:04.070ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
131312025-04-30T06:36:04.071ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
131322025-04-30T06:36:04.071ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
131332025-04-30T06:36:04.123ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
131342025-04-30T06:36:04.123ZDEBGtest_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
131352025-04-30T06:36:04.123ZDEBGtest_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
131362025-04-30T06:36:04.123ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
131372025-04-30T06:36:04.124ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
131382025-04-30T06:36:04.125ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
131392025-04-30T06:36:04.126ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 2b 86 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 +............_de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:62021 req_id = 9fca52ee-f3e9-4280-8442-01fbf97c6d5c
131402025-04-30T06:36:04.126ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 11142,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:62021 req_id = 9fca52ee-f3e9-4280-8442-01fbf97c6d5c
131412025-04-30T06:36:04.126ZDEBGtest_metrics (store): query key key = _dendrite._tcp
131422025-04-30T06:36:04.126ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131432025-04-30T06:36:04.126ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:62021 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9fca52ee-f3e9-4280-8442-01fbf97c6d5c
131442025-04-30T06:36:04.127ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3828, tv_nsec: 926068710 } })
131452025-04-30T06:36:04.127ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
131462025-04-30T06:36:04.127ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f8 4f 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .O..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:63369 req_id = dc220640-de61-45d9-94b4-fce82295f456
131472025-04-30T06:36:04.127ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 63567,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63369 req_id = dc220640-de61-45d9-94b4-fce82295f456
131482025-04-30T06:36:04.128ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131492025-04-30T06:36:04.128ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:63369 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = dc220640-de61-45d9-94b4-fce82295f456
131502025-04-30T06:36:04.130ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
131512025-04-30T06:36:04.241ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
131522025-04-30T06:36:04.241ZDEBGtest_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
131532025-04-30T06:36:04.241ZDEBGtest_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
131542025-04-30T06:36:04.241ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
131552025-04-30T06:36:04.242ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
131562025-04-30T06:36:04.292ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
131572025-04-30T06:36:04.295ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
131582025-04-30T06:36:04.295ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
131592025-04-30T06:36:04.295ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 916.526619ms iteration = 46 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
131602025-04-30T06:36:04.295ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 47 reason = Signaled
131612025-04-30T06:36:04.295ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
131622025-04-30T06:36:04.296ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131632025-04-30T06:36:04.296ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131642025-04-30T06:36:04.296ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131652025-04-30T06:36:04.297ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131662025-04-30T06:36:04.299ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131672025-04-30T06:36:04.309ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
131682025-04-30T06:36:04.309ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
131692025-04-30T06:36:04.309ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
131702025-04-30T06:36:04.310ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
131712025-04-30T06:36:04.330ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
131722025-04-30T06:36:04.330ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
131732025-04-30T06:36:04.332ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: b2 ee 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:51027 req_id = 1f02acaf-3278-4a87-a2fb-47763a2b96bd
131742025-04-30T06:36:04.332ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 45806,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51027 req_id = 1f02acaf-3278-4a87-a2fb-47763a2b96bd
131752025-04-30T06:36:04.332ZDEBGtest_metrics (store): query key key = _mgs._tcp
131762025-04-30T06:36:04.333ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131772025-04-30T06:36:04.333ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:51027 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 1f02acaf-3278-4a87-a2fb-47763a2b96bd
131782025-04-30T06:36:04.333ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3829, tv_nsec: 132893455 } })
131792025-04-30T06:36:04.334ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
131802025-04-30T06:36:04.340ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 43 e4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e C...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58819 req_id = d31087dd-ab7c-4c2e-9a91-b7252d742237
131812025-04-30T06:36:04.340ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17380,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58819 req_id = d31087dd-ab7c-4c2e-9a91-b7252d742237
131822025-04-30T06:36:04.341ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
131832025-04-30T06:36:04.341ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58819 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d31087dd-ab7c-4c2e-9a91-b7252d742237
131842025-04-30T06:36:04.422ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
131852025-04-30T06:36:04.431ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
131862025-04-30T06:36:04.431ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
131872025-04-30T06:36:04.431ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
131882025-04-30T06:36:04.431ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
131892025-04-30T06:36:04.434ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
131902025-04-30T06:36:04.439ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 1e a5 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:60619 req_id = 41efb82c-ba38-4298-aaa3-045e026f77c2
131912025-04-30T06:36:04.440ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 7845,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60619 req_id = 41efb82c-ba38-4298-aaa3-045e026f77c2
131922025-04-30T06:36:04.440ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
131932025-04-30T06:36:04.441ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 7845, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:60619 req_id = 41efb82c-ba38-4298-aaa3-045e026f77c2
131942025-04-30T06:36:04.444ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
131952025-04-30T06:36:04.444ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
131962025-04-30T06:36:04.444ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
131972025-04-30T06:36:04.449ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
131982025-04-30T06:36:04.460ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
131992025-04-30T06:36:04.460ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132002025-04-30T06:36:04.460ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132012025-04-30T06:36:04.461ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132022025-04-30T06:36:04.474ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132032025-04-30T06:36:04.474ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132042025-04-30T06:36:04.475ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132052025-04-30T06:36:04.487ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "2dd23e5e-f960-4741-80ba-34c97780d91f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132062025-04-30T06:36:04.487ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
132072025-04-30T06:36:04.501ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2f3a3976-b3e6-4346-a8b3-a5d7b03e451b", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132082025-04-30T06:36:04.502ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
132092025-04-30T06:36:04.507ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "3aadfb67-8c02-431f-b894-7ae07e6ba570", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132102025-04-30T06:36:04.507ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
132112025-04-30T06:36:04.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132122025-04-30T06:36:04.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132132025-04-30T06:36:04.514ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132142025-04-30T06:36:04.515ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132152025-04-30T06:36:04.515ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132162025-04-30T06:36:04.515ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
132172025-04-30T06:36:04.518ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac09c5e5-fd90-4616-9322-ec8cf80f6a7d", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132182025-04-30T06:36:04.518ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
132192025-04-30T06:36:04.529ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
132202025-04-30T06:36:04.530ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7ecc26fa-5078-485f-bd6e-59d3424cf46e", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132212025-04-30T06:36:04.531ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
132222025-04-30T06:36:04.535ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7b1b8a87-2aed-448b-8576-be6f9978bf31", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132232025-04-30T06:36:04.535ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
132242025-04-30T06:36:04.540ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "479809bc-e859-4bac-abb3-c8eac95fdd43", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132252025-04-30T06:36:04.541ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
132262025-04-30T06:36:04.546ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1513974e-0465-4909-a468-c2d53a21c384", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132272025-04-30T06:36:04.547ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
132282025-04-30T06:36:04.548ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "91f27b3d-818f-446a-bf1f-c8342f9ae5c3", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132292025-04-30T06:36:04.548ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
132302025-04-30T06:36:04.549ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "afa7ccd6-6e9d-457b-afb0-0706d677ab18", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132312025-04-30T06:36:04.550ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
132322025-04-30T06:36:04.553ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "107bc744-734b-45d0-ba58-4f6db382b597", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132332025-04-30T06:36:04.554ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
132342025-04-30T06:36:04.555ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "54e938b7-40f5-4a38-9d43-3bdfb007838c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132352025-04-30T06:36:04.555ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
132362025-04-30T06:36:04.556ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2233f36b-c3fa-4230-9138-3cf3f67bb3f5", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132372025-04-30T06:36:04.557ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
132382025-04-30T06:36:04.561ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "82519c55-2c1e-415c-bf70-7187647513c8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132392025-04-30T06:36:04.561ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
132402025-04-30T06:36:04.564ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "423b132b-0c0b-4946-8c47-399d8261b662", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132412025-04-30T06:36:04.565ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
132422025-04-30T06:36:04.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132432025-04-30T06:36:04.567ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132442025-04-30T06:36:04.568ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
132452025-04-30T06:36:04.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
132462025-04-30T06:36:04.574ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
132472025-04-30T06:36:04.575ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
132482025-04-30T06:36:04.578ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
132492025-04-30T06:36:04.578ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
132502025-04-30T06:36:04.579ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4b75480f-f2fc-4b87-93f9-21fa4a038c52", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132512025-04-30T06:36:04.580ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
132522025-04-30T06:36:04.581ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
132532025-04-30T06:36:04.581ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
132542025-04-30T06:36:04.581ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132552025-04-30T06:36:04.581ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132562025-04-30T06:36:04.582ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132572025-04-30T06:36:04.584ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1cd40230-a0be-489e-ba38-41cc5edacc54", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132582025-04-30T06:36:04.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
132592025-04-30T06:36:04.585ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
132602025-04-30T06:36:04.591ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
132612025-04-30T06:36:04.592ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132622025-04-30T06:36:04.592ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132632025-04-30T06:36:04.592ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132642025-04-30T06:36:04.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8e54a8e-c38b-44bd-aec5-4096a9acd79c", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132652025-04-30T06:36:04.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
132662025-04-30T06:36:04.596ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
132672025-04-30T06:36:04.596ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
132682025-04-30T06:36:04.597ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
132692025-04-30T06:36:04.600ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f22fb968-fffc-4135-86dc-86154f7cf2b1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132702025-04-30T06:36:04.601ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
132712025-04-30T06:36:04.602ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c375421e-eb8e-4439-80a4-bf9d33bda809", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132722025-04-30T06:36:04.602ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
132732025-04-30T06:36:04.604ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4c7aa2cf-4456-4dc0-beb8-1ec56c95034e", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132742025-04-30T06:36:04.604ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
132752025-04-30T06:36:04.607ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4755b2e0-79a1-4a2f-997b-5905cf54b012", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132762025-04-30T06:36:04.608ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
132772025-04-30T06:36:04.611ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b1458568-ab4f-4252-b08f-9ac325d88c3f", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132782025-04-30T06:36:04.611ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
132792025-04-30T06:36:04.613ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8756ec4f-e9dc-42e6-8129-313a9f1bda67", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132802025-04-30T06:36:04.613ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
132812025-04-30T06:36:04.616ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de4782cd-f41b-4650-a7aa-b4cd6fd66388", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132822025-04-30T06:36:04.616ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
132832025-04-30T06:36:04.619ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "891668e5-3516-4bbd-ae2c-ae7fbbaf952b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132842025-04-30T06:36:04.619ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
132852025-04-30T06:36:04.622ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1d725ac0-085f-4fe1-b105-5913f0ba58d2", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132862025-04-30T06:36:04.622ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
132872025-04-30T06:36:04.626ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eb506d01-bd38-423b-b318-6624fc035078", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132882025-04-30T06:36:04.626ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
132892025-04-30T06:36:04.630ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
132902025-04-30T06:36:04.631ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e24644a-0919-473e-b647-18ab95ec4c98", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132912025-04-30T06:36:04.632ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
132922025-04-30T06:36:04.633ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "09cd233f-6419-4d66-8848-88307f857feb", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132932025-04-30T06:36:04.634ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
132942025-04-30T06:36:04.635ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f699c79a-81f4-45f6-bfed-8d8b61f38f6f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132952025-04-30T06:36:04.635ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
132962025-04-30T06:36:04.636ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5e32c7fa-9852-43fe-a14d-4388ebd58067", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132972025-04-30T06:36:04.636ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
132982025-04-30T06:36:04.637ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6debf924-77aa-4837-b059-cc0b79ce8048", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
132992025-04-30T06:36:04.638ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
133002025-04-30T06:36:04.639ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0537a0b0-795d-45e8-a220-3504e3b016f6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133012025-04-30T06:36:04.640ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
133022025-04-30T06:36:04.641ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "85056dda-606d-409b-b26e-ee1844fe6003", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133032025-04-30T06:36:04.642ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
133042025-04-30T06:36:04.644ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a45ad780-1ee0-40a5-a106-638b5977dabe", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133052025-04-30T06:36:04.644ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
133062025-04-30T06:36:04.647ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d13dafdb-5fc7-4a55-b44a-c18187daf417", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133072025-04-30T06:36:04.647ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
133082025-04-30T06:36:04.649ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9316bc62-4a02-42c3-aadf-2e6cfc77a629", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133092025-04-30T06:36:04.650ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
133102025-04-30T06:36:04.653ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a73d3990-9a59-4ee7-b0d0-b9aa4d51284c", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133112025-04-30T06:36:04.653ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
133122025-04-30T06:36:04.659ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf32845c-46ee-42f7-881f-1cd76138c17f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133132025-04-30T06:36:04.660ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
133142025-04-30T06:36:04.662ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b35b1ee7-7b6c-486e-aa85-fd900d3088c2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133152025-04-30T06:36:04.663ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
133162025-04-30T06:36:04.664ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "85da8c35-e49f-4f58-a031-49c7568afd95", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133172025-04-30T06:36:04.664ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
133182025-04-30T06:36:04.665ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "76721d84-c654-4aaf-ba55-2a1e9570d501", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133192025-04-30T06:36:04.665ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
133202025-04-30T06:36:04.667ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "08fadd35-e346-42d3-af13-0dd7dcf41c36", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133212025-04-30T06:36:04.667ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
133222025-04-30T06:36:04.668ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3378cff-02b6-42bb-9989-0b2e64b570ec", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133232025-04-30T06:36:04.669ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133242025-04-30T06:36:04.669ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133252025-04-30T06:36:04.669ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133262025-04-30T06:36:04.669ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133272025-04-30T06:36:04.670ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133282025-04-30T06:36:04.670ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
133292025-04-30T06:36:04.682ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
133302025-04-30T06:36:04.682ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133312025-04-30T06:36:04.682ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133322025-04-30T06:36:04.682ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133332025-04-30T06:36:04.683ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133342025-04-30T06:36:04.687ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133352025-04-30T06:36:04.700ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
133362025-04-30T06:36:04.700ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133372025-04-30T06:36:04.700ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133382025-04-30T06:36:04.701ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133392025-04-30T06:36:04.701ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133402025-04-30T06:36:04.706ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133412025-04-30T06:36:04.706ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
133422025-04-30T06:36:04.706ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133432025-04-30T06:36:04.706ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133442025-04-30T06:36:04.706ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133452025-04-30T06:36:04.707ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133462025-04-30T06:36:04.709ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133472025-04-30T06:36:04.715ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
133482025-04-30T06:36:04.715ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133492025-04-30T06:36:04.715ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133502025-04-30T06:36:04.716ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133512025-04-30T06:36:04.716ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
133522025-04-30T06:36:04.716ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
133532025-04-30T06:36:04.717ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
133542025-04-30T06:36:04.802ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
133552025-04-30T06:36:04.802ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
133562025-04-30T06:36:04.803ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:39922 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
133572025-04-30T06:36:04.805ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:39922 req_id = 8b149a26-7f15-47b5-95ab-a0c409cbdda9 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133582025-04-30T06:36:04.822ZINFOtest_metrics (dropshot): request completed latency_us = 16936 local_addr = [::1]:56237 method = GET remote_addr = [::1]:39922 req_id = 8b149a26-7f15-47b5-95ab-a0c409cbdda9 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
133592025-04-30T06:36:04.823ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "8b149a26-7f15-47b5-95ab-a0c409cbdda9", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
133602025-04-30T06:36:04.831ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133612025-04-30T06:36:04.831ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
133622025-04-30T06:36:04.832ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
133632025-04-30T06:36:04.839ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133642025-04-30T06:36:04.840ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133652025-04-30T06:36:04.840ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133662025-04-30T06:36:04.840ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133672025-04-30T06:36:04.841ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133682025-04-30T06:36:04.854ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133692025-04-30T06:36:04.858ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133702025-04-30T06:36:04.858ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133712025-04-30T06:36:04.858ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133722025-04-30T06:36:04.859ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133732025-04-30T06:36:04.929ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
133742025-04-30T06:36:04.943ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
133752025-04-30T06:36:04.943ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
133762025-04-30T06:36:04.943ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 648.121339ms iteration = 47 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
133772025-04-30T06:36:04.944ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 48 reason = Signaled
133782025-04-30T06:36:04.944ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
133792025-04-30T06:36:04.944ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
133802025-04-30T06:36:04.944ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133812025-04-30T06:36:04.944ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133822025-04-30T06:36:04.944ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133832025-04-30T06:36:04.948ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
133842025-04-30T06:36:04.955ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
133852025-04-30T06:36:04.955ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
133862025-04-30T06:36:04.955ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
133872025-04-30T06:36:04.956ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
133882025-04-30T06:36:04.962ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
133892025-04-30T06:36:04.962ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
133902025-04-30T06:36:04.963ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 92 17 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:58327 req_id = 9bd1a5df-e23f-4415-98bc-bd768e3adfdb
133912025-04-30T06:36:04.963ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 37399,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58327 req_id = 9bd1a5df-e23f-4415-98bc-bd768e3adfdb
133922025-04-30T06:36:04.963ZDEBGtest_metrics (store): query key key = _mgs._tcp
133932025-04-30T06:36:04.963ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
133942025-04-30T06:36:04.964ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58327 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 9bd1a5df-e23f-4415-98bc-bd768e3adfdb
133952025-04-30T06:36:04.964ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3829, tv_nsec: 763303736 } })
133962025-04-30T06:36:04.964ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
133972025-04-30T06:36:04.964ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: fd d8 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53352 req_id = 906dacda-f8bd-4ad1-8471-9965feaaee95
133982025-04-30T06:36:04.964ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 64984,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53352 req_id = 906dacda-f8bd-4ad1-8471-9965feaaee95
133992025-04-30T06:36:04.965ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
134002025-04-30T06:36:04.965ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53352 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 906dacda-f8bd-4ad1-8471-9965feaaee95
134012025-04-30T06:36:05.017ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
134022025-04-30T06:36:05.017ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134032025-04-30T06:36:05.017ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134042025-04-30T06:36:05.017ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134052025-04-30T06:36:05.017ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134062025-04-30T06:36:05.018ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134072025-04-30T06:36:05.018ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 09 fa 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:59613 req_id = eb7b8f72-751f-4193-bf4d-684f1f8f7262
134082025-04-30T06:36:05.018ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 2554,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:59613 req_id = eb7b8f72-751f-4193-bf4d-684f1f8f7262
134092025-04-30T06:36:05.019ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
134102025-04-30T06:36:05.019ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 2554, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:59613 req_id = eb7b8f72-751f-4193-bf4d-684f1f8f7262
134112025-04-30T06:36:05.019ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
134122025-04-30T06:36:05.019ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
134132025-04-30T06:36:05.019ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
134142025-04-30T06:36:05.021ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134152025-04-30T06:36:05.025ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134162025-04-30T06:36:05.026ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134172025-04-30T06:36:05.026ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134182025-04-30T06:36:05.026ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134192025-04-30T06:36:05.028ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "3edbbd86-d16c-428e-89cc-a0a8fa6d79d6", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134202025-04-30T06:36:05.028ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
134212025-04-30T06:36:05.029ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134222025-04-30T06:36:05.029ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134232025-04-30T06:36:05.030ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134242025-04-30T06:36:05.032ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ceb6ad7-dbe2-4864-a9cc-30aec4b2a2fc", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134252025-04-30T06:36:05.032ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
134262025-04-30T06:36:05.034ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1e49ff4f-664d-419f-8fba-357189725421", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134272025-04-30T06:36:05.034ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
134282025-04-30T06:36:05.036ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d542653e-b128-48a8-b86a-49433aae1200", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134292025-04-30T06:36:05.037ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
134302025-04-30T06:36:05.039ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2bd2706-ea0c-487a-a453-5def5b5ac1f7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134312025-04-30T06:36:05.039ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
134322025-04-30T06:36:05.041ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ba73654f-7824-4106-9fe7-be3f2787810d", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134332025-04-30T06:36:05.041ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
134342025-04-30T06:36:05.043ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "83c9c7af-3d36-41e2-912b-086da91db636", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134352025-04-30T06:36:05.044ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
134362025-04-30T06:36:05.046ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ffce15c-5a16-4374-8610-684429da89d5", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134372025-04-30T06:36:05.046ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
134382025-04-30T06:36:05.047ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3666911a-8898-4159-b3dc-e84e7bf59510", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134392025-04-30T06:36:05.047ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
134402025-04-30T06:36:05.049ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "870d0274-5fd0-4fb5-b521-1376e40c1a8c", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134412025-04-30T06:36:05.049ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
134422025-04-30T06:36:05.051ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d8b1061-5149-4289-88ad-8e8bf7438ae4", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134432025-04-30T06:36:05.051ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
134442025-04-30T06:36:05.053ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "7c138983-c390-4277-9dbc-9fa428bb6600", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134452025-04-30T06:36:05.053ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
134462025-04-30T06:36:05.056ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ec2d1ea3-5a86-4440-94b0-9e853b60f3d4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134472025-04-30T06:36:05.056ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
134482025-04-30T06:36:05.067ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
134492025-04-30T06:36:05.072ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c1fc2944-6c22-4e85-96f6-c55f9beb6f65", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134502025-04-30T06:36:05.072ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
134512025-04-30T06:36:05.080ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "175daba3-1ab2-4c7e-a369-ba96879db5bc", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134522025-04-30T06:36:05.081ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
134532025-04-30T06:36:05.095ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c07d94cb-6182-4752-b6c2-d6571855a7dc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134542025-04-30T06:36:05.095ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
134552025-04-30T06:36:05.099ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "275edb63-0b19-4378-b18d-c77c4ad2dab5", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134562025-04-30T06:36:05.099ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
134572025-04-30T06:36:05.103ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "de9d8f8c-0d18-47f4-8351-b11cf6256857", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134582025-04-30T06:36:05.103ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
134592025-04-30T06:36:05.107ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "99d14359-258d-4a28-8c7a-7f4cf3d15915", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134602025-04-30T06:36:05.107ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
134612025-04-30T06:36:05.108ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "61f9311f-16b9-41cd-a00d-8addedda8c00", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134622025-04-30T06:36:05.109ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
134632025-04-30T06:36:05.110ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "98a4bb14-91ea-46d7-ad48-707483e67046", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134642025-04-30T06:36:05.110ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
134652025-04-30T06:36:05.111ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ebe76d88-4d4f-4178-bfba-0a54e9f6583a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134662025-04-30T06:36:05.111ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
134672025-04-30T06:36:05.113ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e72e00bf-c351-4f07-836d-2c4be5d85869", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134682025-04-30T06:36:05.113ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
134692025-04-30T06:36:05.115ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "0c83d3a3-1c28-45b1-bab5-4c586a55155d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134702025-04-30T06:36:05.115ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
134712025-04-30T06:36:05.118ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "7d0a85ca-3209-4149-8c25-cf52ca3d2015", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134722025-04-30T06:36:05.118ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
134732025-04-30T06:36:05.119ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
134742025-04-30T06:36:05.119ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
134752025-04-30T06:36:05.119ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134762025-04-30T06:36:05.119ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134772025-04-30T06:36:05.120ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134782025-04-30T06:36:05.123ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
134792025-04-30T06:36:05.127ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
134802025-04-30T06:36:05.127ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134812025-04-30T06:36:05.127ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134822025-04-30T06:36:05.128ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134832025-04-30T06:36:05.128ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ac739a85-c3d3-471c-b2a9-6df39eb20fcb", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134842025-04-30T06:36:05.128ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
134852025-04-30T06:36:05.131ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
134862025-04-30T06:36:05.131ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
134872025-04-30T06:36:05.131ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
134882025-04-30T06:36:05.133ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "891d2673-01d0-45a6-afca-ef16981a0616", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134892025-04-30T06:36:05.133ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
134902025-04-30T06:36:05.137ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94a0386a-0abf-4e0d-becb-4af63e149b3d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134912025-04-30T06:36:05.137ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
134922025-04-30T06:36:05.141ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee122c29-331e-41b9-94e4-9c1a54fa6180", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134932025-04-30T06:36:05.141ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
134942025-04-30T06:36:05.146ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "92c9a422-40e8-487b-bff0-c3953cd86c1f", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134952025-04-30T06:36:05.146ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
134962025-04-30T06:36:05.148ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e41922ab-cfc8-4006-a0e6-f85820a684ca", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134972025-04-30T06:36:05.148ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
134982025-04-30T06:36:05.149ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "966d2e97-ffa5-4191-901f-2603d374c397", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
134992025-04-30T06:36:05.150ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
135002025-04-30T06:36:05.151ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "39123be0-8268-45c6-9a5c-d0c4b183e727", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135012025-04-30T06:36:05.152ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
135022025-04-30T06:36:05.154ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b29c3879-9455-4c94-915a-1af2235156a9", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135032025-04-30T06:36:05.154ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
135042025-04-30T06:36:05.159ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3aaf309-5cc1-44fa-aceb-c18ca456899e", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135052025-04-30T06:36:05.159ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
135062025-04-30T06:36:05.160ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
135072025-04-30T06:36:05.166ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "af06fa4f-3ba2-4f44-848a-89dbf9a5960f", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135082025-04-30T06:36:05.166ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
135092025-04-30T06:36:05.172ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "1b763bc7-7f80-46ef-bdde-558f4cddd14b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135102025-04-30T06:36:05.172ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
135112025-04-30T06:36:05.180ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5b2e3b32-e419-44cd-92ca-d1a2ea0f17b9", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135122025-04-30T06:36:05.180ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
135132025-04-30T06:36:05.184ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4071240d-29e4-46fb-b144-b8138ff0492f", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135142025-04-30T06:36:05.185ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
135152025-04-30T06:36:05.190ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e37ba010-8674-4ae2-8109-8bbc2582a61b", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135162025-04-30T06:36:05.190ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
135172025-04-30T06:36:05.196ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "74ccfd44-d30d-42ba-a165-aa8079da8af1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135182025-04-30T06:36:05.196ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
135192025-04-30T06:36:05.197ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "dae3efb5-8271-4610-a1e9-29ef92c9c51a", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135202025-04-30T06:36:05.197ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
135212025-04-30T06:36:05.200ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e9fc4f1d-0854-4910-bddc-85389e83107a", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135222025-04-30T06:36:05.204ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
135232025-04-30T06:36:05.206ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a853e806-75f0-4018-a345-a0a088301290", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135242025-04-30T06:36:05.206ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
135252025-04-30T06:36:05.207ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2005569a-885f-4acc-ba46-8e35ea090441", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135262025-04-30T06:36:05.208ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135272025-04-30T06:36:05.208ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135282025-04-30T06:36:05.208ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135292025-04-30T06:36:05.208ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135302025-04-30T06:36:05.209ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135312025-04-30T06:36:05.212ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135322025-04-30T06:36:05.224ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135332025-04-30T06:36:05.224ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
135342025-04-30T06:36:05.224ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135352025-04-30T06:36:05.224ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135362025-04-30T06:36:05.224ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135372025-04-30T06:36:05.225ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135382025-04-30T06:36:05.226ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
135392025-04-30T06:36:05.226ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135402025-04-30T06:36:05.226ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135412025-04-30T06:36:05.226ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
135422025-04-30T06:36:05.227ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135432025-04-30T06:36:05.230ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135442025-04-30T06:36:05.236ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
135452025-04-30T06:36:05.236ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135462025-04-30T06:36:05.236ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135472025-04-30T06:36:05.236ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135482025-04-30T06:36:05.237ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135492025-04-30T06:36:05.250ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
135502025-04-30T06:36:05.250ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135512025-04-30T06:36:05.250ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
135522025-04-30T06:36:05.251ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135532025-04-30T06:36:05.316ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
135542025-04-30T06:36:05.316ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
135552025-04-30T06:36:05.317ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
135562025-04-30T06:36:05.317ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
135572025-04-30T06:36:05.318ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
135582025-04-30T06:36:05.318ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:41261 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
135592025-04-30T06:36:05.319ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
135602025-04-30T06:36:05.320ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135612025-04-30T06:36:05.321ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ce3e7bcb-d58f-4052-ba6e-c9bd03eb7b4f uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135622025-04-30T06:36:05.321ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:41261 req_id = 07ff7459-d965-4170-9177-3bf4850f042a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135632025-04-30T06:36:05.321ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ce3e7bcb-d58f-4052-ba6e-c9bd03eb7b4f resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135642025-04-30T06:36:05.321ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ce3e7bcb-d58f-4052-ba6e-c9bd03eb7b4f roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135652025-04-30T06:36:05.322ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ce3e7bcb-d58f-4052-ba6e-c9bd03eb7b4f resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135662025-04-30T06:36:05.323ZINFOtest_metrics (dropshot): request completed latency_us = 2088 local_addr = [::1]:56237 method = GET remote_addr = [::1]:41261 req_id = 07ff7459-d965-4170-9177-3bf4850f042a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
135672025-04-30T06:36:05.323ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "07ff7459-d965-4170-9177-3bf4850f042a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135682025-04-30T06:36:05.326ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135692025-04-30T06:36:05.326ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
135702025-04-30T06:36:05.326ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
135712025-04-30T06:36:05.328ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135722025-04-30T06:36:05.328ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
135732025-04-30T06:36:05.329ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135742025-04-30T06:36:05.329ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135752025-04-30T06:36:05.329ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135762025-04-30T06:36:05.332ZINFOtest_metrics (dropshot_internal): request completed latency_us = 11647 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = ce3e7bcb-d58f-4052-ba6e-c9bd03eb7b4f response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
135772025-04-30T06:36:05.335ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce3e7bcb-d58f-4052-ba6e-c9bd03eb7b4f", "content-length": "405", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135782025-04-30T06:36:05.336ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135792025-04-30T06:36:05.336ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
135802025-04-30T06:36:05.340ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
135812025-04-30T06:36:05.340ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
135822025-04-30T06:36:05.340ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
135832025-04-30T06:36:05.341ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
135842025-04-30T06:36:05.343ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3d562dcb-144b-4513-a74e-ca3f89108bbd uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135852025-04-30T06:36:05.344ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3d562dcb-144b-4513-a74e-ca3f89108bbd resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135862025-04-30T06:36:05.344ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3d562dcb-144b-4513-a74e-ca3f89108bbd roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135872025-04-30T06:36:05.344ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3d562dcb-144b-4513-a74e-ca3f89108bbd resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135882025-04-30T06:36:05.352ZINFOtest_metrics (dropshot_internal): request completed latency_us = 9028 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 3d562dcb-144b-4513-a74e-ca3f89108bbd response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
135892025-04-30T06:36:05.354ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "3d562dcb-144b-4513-a74e-ca3f89108bbd", "content-length": "29", "date": "Wed, 30 Apr 2025 06:36:04 GMT"} })
135902025-04-30T06:36:05.354ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
135912025-04-30T06:36:05.354ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
135922025-04-30T06:36:05.354ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
135932025-04-30T06:36:05.396ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
135942025-04-30T06:36:05.414ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
135952025-04-30T06:36:05.415ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
135962025-04-30T06:36:05.415ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 471.395455ms iteration = 48 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
135972025-04-30T06:36:05.415ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 49 reason = Signaled
135982025-04-30T06:36:05.415ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
135992025-04-30T06:36:05.415ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136002025-04-30T06:36:05.415ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136012025-04-30T06:36:05.415ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136022025-04-30T06:36:05.416ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136032025-04-30T06:36:05.418ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136042025-04-30T06:36:05.423ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
136052025-04-30T06:36:05.423ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
136062025-04-30T06:36:05.423ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
136072025-04-30T06:36:05.424ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
136082025-04-30T06:36:05.427ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
136092025-04-30T06:36:05.427ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
136102025-04-30T06:36:05.428ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: e4 51 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 .Q..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:61037 req_id = ed03f4d4-7fcb-4e0e-8f35-71b9349763d1
136112025-04-30T06:36:05.428ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 58449,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:61037 req_id = ed03f4d4-7fcb-4e0e-8f35-71b9349763d1
136122025-04-30T06:36:05.428ZDEBGtest_metrics (store): query key key = _mgs._tcp
136132025-04-30T06:36:05.428ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136142025-04-30T06:36:05.428ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:61037 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = ed03f4d4-7fcb-4e0e-8f35-71b9349763d1
136152025-04-30T06:36:05.428ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3830, tv_nsec: 227889623 } })
136162025-04-30T06:36:05.429ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
136172025-04-30T06:36:05.429ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 50 87 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e P...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60439 req_id = f923cfec-7c43-4eae-aa8d-78dadbfe1f47
136182025-04-30T06:36:05.429ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 20615,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60439 req_id = f923cfec-7c43-4eae-aa8d-78dadbfe1f47
136192025-04-30T06:36:05.429ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
136202025-04-30T06:36:05.429ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60439 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = f923cfec-7c43-4eae-aa8d-78dadbfe1f47
136212025-04-30T06:36:05.495ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
136222025-04-30T06:36:05.496ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
136232025-04-30T06:36:05.496ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
136242025-04-30T06:36:05.496ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136252025-04-30T06:36:05.496ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
136262025-04-30T06:36:05.497ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136272025-04-30T06:36:05.500ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 6e a8 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c n............_cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53077 req_id = 33bc1dad-1c62-4970-bcd5-400e35f83ec0
136282025-04-30T06:36:05.500ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 28328,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53077 req_id = 33bc1dad-1c62-4970-bcd5-400e35f83ec0
136292025-04-30T06:36:05.500ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
136302025-04-30T06:36:05.501ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 28328, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53077 req_id = 33bc1dad-1c62-4970-bcd5-400e35f83ec0
136312025-04-30T06:36:05.501ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
136322025-04-30T06:36:05.506ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
136332025-04-30T06:36:05.507ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136342025-04-30T06:36:05.507ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
136352025-04-30T06:36:05.508ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136362025-04-30T06:36:05.508ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
136372025-04-30T06:36:05.509ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
136382025-04-30T06:36:05.509ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
136392025-04-30T06:36:05.512ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136402025-04-30T06:36:05.512ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136412025-04-30T06:36:05.513ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
136422025-04-30T06:36:05.513ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
136432025-04-30T06:36:05.514ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
136442025-04-30T06:36:05.516ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "06b31cd8-e497-4e62-a357-08c3d550330f", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136452025-04-30T06:36:05.516ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
136462025-04-30T06:36:05.516ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136472025-04-30T06:36:05.517ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136482025-04-30T06:36:05.517ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136492025-04-30T06:36:05.517ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
136502025-04-30T06:36:05.519ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f0b3600-b063-4ef2-91b7-07fabb5a8c6c", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136512025-04-30T06:36:05.519ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
136522025-04-30T06:36:05.522ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "df53dcd8-ba9d-4072-a6f0-d6a95a0a0c01", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136532025-04-30T06:36:05.522ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
136542025-04-30T06:36:05.524ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a617c5f6-7a96-4422-aa3a-8d38b3486af1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136552025-04-30T06:36:05.524ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
136562025-04-30T06:36:05.527ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "edbcac64-ba5c-40f3-85a8-540a3d2db45f", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136572025-04-30T06:36:05.527ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
136582025-04-30T06:36:05.533ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c78f2b48-9948-42d0-8f6a-aa0da510a0b6", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136592025-04-30T06:36:05.533ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
136602025-04-30T06:36:05.538ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8ad81d9a-3322-4d32-acd1-18b6e7483627", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136612025-04-30T06:36:05.538ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
136622025-04-30T06:36:05.542ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cc804a4f-86a5-43f1-acf2-65c85f7088de", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136632025-04-30T06:36:05.542ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
136642025-04-30T06:36:05.546ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0379bff4-287f-4454-8c0f-f8612349e587", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136652025-04-30T06:36:05.546ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
136662025-04-30T06:36:05.547ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3668592-a6fe-42b8-bd64-1acab27e98a9", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136672025-04-30T06:36:05.548ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
136682025-04-30T06:36:05.558ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
136692025-04-30T06:36:05.560ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:63396 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
136702025-04-30T06:36:05.560ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ab64641-d9b6-4811-a226-5bc2ea6c981b", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136712025-04-30T06:36:05.561ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
136722025-04-30T06:36:05.562ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:63396 req_id = c9033c33-9171-4f8c-8ea5-6ad670bc5a76 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
136732025-04-30T06:36:05.562ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 523 local_addr = [::1]:38907 method = GET remote_addr = [::1]:63396 req_id = c9033c33-9171-4f8c-8ea5-6ad670bc5a76 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
136742025-04-30T06:36:05.563ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6903295a-f2cc-49ac-824b-c6fe7f723c28", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136752025-04-30T06:36:05.564ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
136762025-04-30T06:36:05.566ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "08732536-d360-4ea8-9dc7-700d0589ace4", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136772025-04-30T06:36:05.566ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
136782025-04-30T06:36:05.567ZDEBGtest_metrics (results-sink): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136792025-04-30T06:36:05.567ZDEBGtest_metrics (results-sink-cluster): inserting 1 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136802025-04-30T06:36:05.567ZDEBGtest_metrics (clickhouse-client): unrolling 1 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
136812025-04-30T06:36:05.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
136822025-04-30T06:36:05.573ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8c5fc1ba-6e97-412c-8556-f9da20bb0edd", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136832025-04-30T06:36:05.574ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
136842025-04-30T06:36:05.577ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
136852025-04-30T06:36:05.577ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 1 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
136862025-04-30T06:36:05.581ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 1 table_name = measurements_i64
136872025-04-30T06:36:05.581ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
136882025-04-30T06:36:05.583ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "3849c577-74cd-4adf-b1b3-0664d35be740", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136892025-04-30T06:36:05.583ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
136902025-04-30T06:36:05.585ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "8bd1f4b6-622f-4062-9383-97ce83f96ab2", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136912025-04-30T06:36:05.585ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
136922025-04-30T06:36:05.586ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "d3fb8c6c-0996-462e-a7a4-fc1ad3ddbc83", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136932025-04-30T06:36:05.586ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
136942025-04-30T06:36:05.588ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a194d0d6-1864-4170-a99f-968f37b55db0", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136952025-04-30T06:36:05.588ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
136962025-04-30T06:36:05.590ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "0bdbc260-c360-4556-8bbb-da344d44013b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136972025-04-30T06:36:05.590ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
136982025-04-30T06:36:05.591ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "83a2bad2-b6ca-48c4-a176-93ebe8c2cd5f", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
136992025-04-30T06:36:05.591ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
137002025-04-30T06:36:05.592ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "6c30e76e-dc2b-4017-906f-d05a10385c75", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137012025-04-30T06:36:05.593ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
137022025-04-30T06:36:05.594ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e3c30a42-5e15-4fb4-a0bb-3ac64cabd1fe", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137032025-04-30T06:36:05.594ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
137042025-04-30T06:36:05.595ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "5ab9eee2-eecb-4e9d-a92e-a554b2979c1c", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137052025-04-30T06:36:05.595ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
137062025-04-30T06:36:05.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2d2bef9e-62e8-49ea-8e5c-a20aed4ca53a", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137072025-04-30T06:36:05.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
137082025-04-30T06:36:05.601ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1f51c425-31d9-4796-8488-5ccc9a105e43", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137092025-04-30T06:36:05.601ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
137102025-04-30T06:36:05.604ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "ad950548-13fa-4691-b3e4-5a0b76c07df4", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137112025-04-30T06:36:05.604ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
137122025-04-30T06:36:05.607ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19e439d8-5c60-406e-af7b-758acfa08cd0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137132025-04-30T06:36:05.607ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
137142025-04-30T06:36:05.609ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137152025-04-30T06:36:05.609ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137162025-04-30T06:36:05.609ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137172025-04-30T06:36:05.609ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137182025-04-30T06:36:05.611ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137192025-04-30T06:36:05.615ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a2735a43-252c-4060-8396-723ad8981e7d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137202025-04-30T06:36:05.615ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
137212025-04-30T06:36:05.617ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137222025-04-30T06:36:05.634ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
137232025-04-30T06:36:05.634ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137242025-04-30T06:36:05.634ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137252025-04-30T06:36:05.635ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137262025-04-30T06:36:05.636ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d81288f9-5a6c-41f2-be58-e22512f2c665", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137272025-04-30T06:36:05.636ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
137282025-04-30T06:36:05.638ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137292025-04-30T06:36:05.638ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137302025-04-30T06:36:05.638ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137312025-04-30T06:36:05.641ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a8a53998-6328-4a52-8ee9-9b917fe6b57c", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137322025-04-30T06:36:05.641ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
137332025-04-30T06:36:05.643ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "808ab491-8277-430a-93f0-89891d11404c", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137342025-04-30T06:36:05.643ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
137352025-04-30T06:36:05.645ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "465a4a0b-0cf8-4686-84d2-8eae0a523167", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137362025-04-30T06:36:05.645ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
137372025-04-30T06:36:05.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0dbd72b-4eef-4d47-b023-f0f6b20ef2d5", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137382025-04-30T06:36:05.646ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
137392025-04-30T06:36:05.648ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a95b1c2b-f115-4704-80fb-04cda3714cf0", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137402025-04-30T06:36:05.648ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
137412025-04-30T06:36:05.649ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a93c5fb4-177e-46d4-aad2-081c32a7e10d", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137422025-04-30T06:36:05.650ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
137432025-04-30T06:36:05.655ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bb22e349-4885-4b5d-90b8-20ac5f49d01b", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137442025-04-30T06:36:05.656ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
137452025-04-30T06:36:05.660ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cd478df6-80f1-43d6-b568-3f849f142468", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137462025-04-30T06:36:05.660ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
137472025-04-30T06:36:05.663ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "e77af3b2-3adb-4e2f-ab36-bc02225d9293", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137482025-04-30T06:36:05.663ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
137492025-04-30T06:36:05.666ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b0dc58b6-7402-4801-ad50-e2a3b662b1c0", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137502025-04-30T06:36:05.666ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
137512025-04-30T06:36:05.669ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
137522025-04-30T06:36:05.672ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
137532025-04-30T06:36:05.675ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "107dc5b8-9eb8-4829-a952-addd62450a92", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137542025-04-30T06:36:05.675ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
137552025-04-30T06:36:05.678ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "fe12eaf8-f3e3-4a64-aced-022d20924ae3", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137562025-04-30T06:36:05.678ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
137572025-04-30T06:36:05.679ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "79074f04-ba02-4f2d-991b-92a4e9f43ea6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137582025-04-30T06:36:05.679ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
137592025-04-30T06:36:05.681ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f224dcb7-ff5a-4ae9-a345-708ec6a4c1d2", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137602025-04-30T06:36:05.681ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
137612025-04-30T06:36:05.681ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c8eec2b0-78a0-4c21-aba9-e4da84ee3afb", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137622025-04-30T06:36:05.681ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
137632025-04-30T06:36:05.682ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b4aaa39b-d92a-4f6d-aa8d-55c3fc0b4d37", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137642025-04-30T06:36:05.682ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137652025-04-30T06:36:05.683ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137662025-04-30T06:36:05.683ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137672025-04-30T06:36:05.683ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137682025-04-30T06:36:05.684ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137692025-04-30T06:36:05.686ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137702025-04-30T06:36:05.690ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137712025-04-30T06:36:05.690ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
137722025-04-30T06:36:05.690ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137732025-04-30T06:36:05.690ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137742025-04-30T06:36:05.690ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137752025-04-30T06:36:05.691ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137762025-04-30T06:36:05.693ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
137772025-04-30T06:36:05.696ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
137782025-04-30T06:36:05.697ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137792025-04-30T06:36:05.697ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137802025-04-30T06:36:05.697ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
137812025-04-30T06:36:05.803ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
137822025-04-30T06:36:05.803ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
137832025-04-30T06:36:05.805ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
137842025-04-30T06:36:05.806ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137852025-04-30T06:36:05.806ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
137862025-04-30T06:36:05.806ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
137872025-04-30T06:36:05.808ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
137882025-04-30T06:36:05.808ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:47985 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
137892025-04-30T06:36:05.809ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:47985 req_id = 0d9613bc-2dce-4980-93a1-3b21fb1e916a sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137902025-04-30T06:36:05.810ZINFOtest_metrics (dropshot): request completed latency_us = 890 local_addr = [::1]:56237 method = GET remote_addr = [::1]:47985 req_id = 0d9613bc-2dce-4980-93a1-3b21fb1e916a response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
137912025-04-30T06:36:05.811ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "0d9613bc-2dce-4980-93a1-3b21fb1e916a", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
137922025-04-30T06:36:05.814ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137932025-04-30T06:36:05.814ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
137942025-04-30T06:36:05.814ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
137952025-04-30T06:36:05.817ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
137962025-04-30T06:36:05.818ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
137972025-04-30T06:36:05.818ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
137982025-04-30T06:36:05.818ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
137992025-04-30T06:36:05.818ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138002025-04-30T06:36:05.819ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138012025-04-30T06:36:05.825ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138022025-04-30T06:36:05.825ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138032025-04-30T06:36:05.825ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138042025-04-30T06:36:05.825ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138052025-04-30T06:36:05.827ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138062025-04-30T06:36:05.832ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138072025-04-30T06:36:05.832ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138082025-04-30T06:36:05.832ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138092025-04-30T06:36:05.833ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138102025-04-30T06:36:05.833ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138112025-04-30T06:36:05.833ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138122025-04-30T06:36:05.834ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138132025-04-30T06:36:05.960ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
138142025-04-30T06:36:05.960ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
138152025-04-30T06:36:05.961ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 546.306219ms iteration = 49 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
138162025-04-30T06:36:05.961ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 50 reason = Signaled
138172025-04-30T06:36:05.961ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
138182025-04-30T06:36:05.961ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138192025-04-30T06:36:05.962ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138202025-04-30T06:36:05.962ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138212025-04-30T06:36:05.962ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138222025-04-30T06:36:05.975ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138232025-04-30T06:36:05.981ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
138242025-04-30T06:36:05.981ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
138252025-04-30T06:36:05.981ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
138262025-04-30T06:36:05.981ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
138272025-04-30T06:36:05.986ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
138282025-04-30T06:36:05.986ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
138292025-04-30T06:36:05.986ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
138302025-04-30T06:36:05.988ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 61 56 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 aV..........._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:63015 req_id = f94f76f2-cd51-4386-8e8b-6f26d609191e
138312025-04-30T06:36:05.988ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 24918,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:63015 req_id = f94f76f2-cd51-4386-8e8b-6f26d609191e
138322025-04-30T06:36:05.988ZDEBGtest_metrics (store): query key key = _mgs._tcp
138332025-04-30T06:36:05.988ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138342025-04-30T06:36:05.988ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:63015 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = f94f76f2-cd51-4386-8e8b-6f26d609191e
138352025-04-30T06:36:05.989ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3830, tv_nsec: 787986770 } })
138362025-04-30T06:36:05.989ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
138372025-04-30T06:36:05.989ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 79 75 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e yu..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:53311 req_id = c1d795fb-5f5c-4010-a6eb-6ac8e01f9f3a
138382025-04-30T06:36:05.989ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31093,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53311 req_id = c1d795fb-5f5c-4010-a6eb-6ac8e01f9f3a
138392025-04-30T06:36:05.989ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138402025-04-30T06:36:05.989ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:53311 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = c1d795fb-5f5c-4010-a6eb-6ac8e01f9f3a
138412025-04-30T06:36:06.061ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
138422025-04-30T06:36:06.061ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
138432025-04-30T06:36:06.061ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
138442025-04-30T06:36:06.061ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138452025-04-30T06:36:06.061ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138462025-04-30T06:36:06.062ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138472025-04-30T06:36:06.062ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 58 34 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c X4..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:58443 req_id = 96774386-b396-4089-9420-2d57c91ecbe1
138482025-04-30T06:36:06.062ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 22580,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58443 req_id = 96774386-b396-4089-9420-2d57c91ecbe1
138492025-04-30T06:36:06.062ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
138502025-04-30T06:36:06.063ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 22580, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:58443 req_id = 96774386-b396-4089-9420-2d57c91ecbe1
138512025-04-30T06:36:06.063ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
138522025-04-30T06:36:06.063ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
138532025-04-30T06:36:06.063ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
138542025-04-30T06:36:06.064ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
138552025-04-30T06:36:06.069ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
138562025-04-30T06:36:06.069ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138572025-04-30T06:36:06.069ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138582025-04-30T06:36:06.070ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138592025-04-30T06:36:06.071ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "e429bc76-5af3-4c21-a485-7ba2cc5f4cdb", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
138602025-04-30T06:36:06.071ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
138612025-04-30T06:36:06.072ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
138622025-04-30T06:36:06.073ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: e1 32 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .2..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:58610 req_id = d9f7e18b-6305-481b-a756-f7e484a63310
138632025-04-30T06:36:06.073ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 57650,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58610 req_id = d9f7e18b-6305-481b-a756-f7e484a63310
138642025-04-30T06:36:06.073ZDEBGtest_metrics (store): query key key = _dendrite._tcp
138652025-04-30T06:36:06.074ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138662025-04-30T06:36:06.074ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:58610 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = d9f7e18b-6305-481b-a756-f7e484a63310
138672025-04-30T06:36:06.074ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
138682025-04-30T06:36:06.074ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
138692025-04-30T06:36:06.074ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
138702025-04-30T06:36:06.074ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3830, tv_nsec: 873644958 } })
138712025-04-30T06:36:06.074ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
138722025-04-30T06:36:06.075ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 79 38 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e y8..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:64635 req_id = 5d35da65-a635-43b8-96a9-4323059b8c39
138732025-04-30T06:36:06.075ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31032,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64635 req_id = 5d35da65-a635-43b8-96a9-4323059b8c39
138742025-04-30T06:36:06.075ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
138752025-04-30T06:36:06.076ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:64635 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 5d35da65-a635-43b8-96a9-4323059b8c39
138762025-04-30T06:36:06.076ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d2a8f490-504c-438b-b350-db0a4f151dea", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:05 GMT"} })
138772025-04-30T06:36:06.076ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
138782025-04-30T06:36:06.078ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
138792025-04-30T06:36:06.137ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
138802025-04-30T06:36:06.137ZDEBGtest_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
138812025-04-30T06:36:06.138ZDEBGtest_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
138822025-04-30T06:36:06.138ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
138832025-04-30T06:36:06.139ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
138842025-04-30T06:36:06.141ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
138852025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:55308 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138862025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:57213 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138872025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:48953 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138882025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:64928 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138892025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:43204 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138902025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:63638 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138912025-04-30T06:36:06.649ZINFOtest_metrics (dropshot): accepted connection kind = pantry local_addr = [::1]:38907 remote_addr = [::1]:54353 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
138922025-04-30T06:36:06.649ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
138932025-04-30T06:36:06.650ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
138942025-04-30T06:36:06.651ZTRACtest_metrics (results-sink): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
138952025-04-30T06:36:06.651ZTRACtest_metrics (results-sink-cluster): batch interval expired, but no samples to insert collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
138962025-04-30T06:36:06.651ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 33 40 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 3@..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:64743 req_id = 8914160f-8176-453d-8d40-1174d3e64774
138972025-04-30T06:36:06.651ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 13120,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:64743 req_id = 8914160f-8176-453d-8d40-1174d3e64774
138982025-04-30T06:36:06.654ZDEBGtest_metrics (store): query key key = _dendrite._tcp
138992025-04-30T06:36:06.655ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139002025-04-30T06:36:06.655ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:64743 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 8914160f-8176-453d-8d40-1174d3e64774
139012025-04-30T06:36:06.659ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139022025-04-30T06:36:06.662ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3831, tv_nsec: 461262859 } })
139032025-04-30T06:36:06.662ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
139042025-04-30T06:36:06.662ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:55308 req_id = 0e39ad09-cb20-4298-a794-8eb5acf934c4 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139052025-04-30T06:36:06.662ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:57213 req_id = d133b02c-e4f3-4343-ae20-b15f1c441010 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139062025-04-30T06:36:06.663ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:48953 req_id = 6b597a35-f7bd-41b2-89fe-aa506c7ca017 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139072025-04-30T06:36:06.663ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:64928 req_id = a8f6a896-a148-4fae-a9e1-7d19ede3a8f2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139082025-04-30T06:36:06.663ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:43204 req_id = bc2ddc75-4852-4500-a83d-f508ab733e2b sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139092025-04-30T06:36:06.663ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:63638 req_id = b18d803b-0993-4619-9c74-8038969d13c2 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139102025-04-30T06:36:06.664ZTRACtest_metrics (dropshot): incoming request kind = pantry local_addr = [::1]:38907 method = GET remote_addr = [::1]:54353 req_id = e9f0c3eb-4336-4de0-920a-94b7b0089098 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139112025-04-30T06:36:06.666ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3430 local_addr = [::1]:38907 method = GET remote_addr = [::1]:55308 req_id = 0e39ad09-cb20-4298-a794-8eb5acf934c4 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139122025-04-30T06:36:06.666ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3466 local_addr = [::1]:38907 method = GET remote_addr = [::1]:57213 req_id = d133b02c-e4f3-4343-ae20-b15f1c441010 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139132025-04-30T06:36:06.666ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3370 local_addr = [::1]:38907 method = GET remote_addr = [::1]:48953 req_id = 6b597a35-f7bd-41b2-89fe-aa506c7ca017 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139142025-04-30T06:36:06.666ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3531 local_addr = [::1]:38907 method = GET remote_addr = [::1]:64928 req_id = a8f6a896-a148-4fae-a9e1-7d19ede3a8f2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139152025-04-30T06:36:06.666ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3413 local_addr = [::1]:38907 method = GET remote_addr = [::1]:43204 req_id = bc2ddc75-4852-4500-a83d-f508ab733e2b response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139162025-04-30T06:36:06.667ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3390 local_addr = [::1]:38907 method = GET remote_addr = [::1]:63638 req_id = b18d803b-0993-4619-9c74-8038969d13c2 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139172025-04-30T06:36:06.667ZINFOtest_metrics (dropshot): request completed kind = pantry latency_us = 3311 local_addr = [::1]:38907 method = GET remote_addr = [::1]:54353 req_id = e9f0c3eb-4336-4de0-920a-94b7b0089098 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /crucible/pantry/0
139182025-04-30T06:36:06.669ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: f5 78 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 .x..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:54105 req_id = 6f633382-a589-4646-95be-bbb8c18c5408
139192025-04-30T06:36:06.669ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62840,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:54105 req_id = 6f633382-a589-4646-95be-bbb8c18c5408
139202025-04-30T06:36:06.669ZDEBGtest_metrics (store): query key key = _dendrite._tcp
139212025-04-30T06:36:06.670ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139222025-04-30T06:36:06.670ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:54105 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 6f633382-a589-4646-95be-bbb8c18c5408
139232025-04-30T06:36:06.671ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139242025-04-30T06:36:06.672ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139252025-04-30T06:36:06.672ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139262025-04-30T06:36:06.672ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
139272025-04-30T06:36:06.675ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3831, tv_nsec: 473986630 } })
139282025-04-30T06:36:06.675ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
139292025-04-30T06:36:06.676ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 8f c4 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:57828 req_id = 69179b8e-9e64-417f-9ebc-d139a97446f0
139302025-04-30T06:36:06.676ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 36804,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:57828 req_id = 69179b8e-9e64-417f-9ebc-d139a97446f0
139312025-04-30T06:36:06.678ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139322025-04-30T06:36:06.678ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:57828 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 69179b8e-9e64-417f-9ebc-d139a97446f0
139332025-04-30T06:36:06.680ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = bfd_manager
139342025-04-30T06:36:06.758ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = bfd_manager zone_address = ::1
139352025-04-30T06:36:06.759ZDEBGtest_metrics (MgsClient): client request background_task = bfd_manager body = None method = GET uri = http://[::1]:12225/local/switch-id
139362025-04-30T06:36:06.759ZDEBGtest_metrics (MgsClient): client response background_task = bfd_manager result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139372025-04-30T06:36:06.759ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = bfd_manager reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
139382025-04-30T06:36:06.761ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = bfd_manager
139392025-04-30T06:36:06.764ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: f5 1a 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:51894 req_id = 3cdddc7a-2f41-4092-abe9-7c7b6f1a7072
139402025-04-30T06:36:06.764ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 62746,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:51894 req_id = 3cdddc7a-2f41-4092-abe9-7c7b6f1a7072
139412025-04-30T06:36:06.764ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
139422025-04-30T06:36:06.765ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:51894 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 3cdddc7a-2f41-4092-abe9-7c7b6f1a7072
139432025-04-30T06:36:06.765ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
139442025-04-30T06:36:06.812ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc zone_address = ::1
139452025-04-30T06:36:06.812ZDEBGtest_metrics (MgsClient): client request background_task = switch_port_config_manager body = None method = GET rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc uri = http://[::1]:12225/local/switch-id
139462025-04-30T06:36:06.812ZDEBGtest_metrics (MgsClient): client response background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
139472025-04-30T06:36:06.812ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
139482025-04-30T06:36:06.814ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = switch_port_config_manager rack_id = c19a698f-c6f9-4a17-ae30-20d711b8f7dc
139492025-04-30T06:36:06.870ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5bbb2170-be87-430b-8b7f-636914b1e600", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139502025-04-30T06:36:06.870ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
139512025-04-30T06:36:06.878ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
139522025-04-30T06:36:06.883ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a41c2a02-ce70-44dc-892a-32603f452e22", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139532025-04-30T06:36:06.883ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
139542025-04-30T06:36:06.887ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "dea78611-8bc8-4ded-9424-526156b5f4ad", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139552025-04-30T06:36:06.887ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
139562025-04-30T06:36:06.897ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2ee57e5e-5f48-4c8c-be4d-10aec3aa6830", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139572025-04-30T06:36:06.897ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
139582025-04-30T06:36:06.900ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "160978fa-a8fb-4d17-82e6-eaf753d7ba90", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139592025-04-30T06:36:06.900ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
139602025-04-30T06:36:06.904ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "06aca15a-7603-4c89-8c3e-c04fdb11d830", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139612025-04-30T06:36:06.904ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
139622025-04-30T06:36:06.905ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "77867344-7018-4bbe-976e-edd8b6a073d8", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139632025-04-30T06:36:06.905ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
139642025-04-30T06:36:06.907ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "af02086f-56b3-4c5a-8279-e24ebf6fd820", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139652025-04-30T06:36:06.907ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
139662025-04-30T06:36:06.908ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "43f2f02c-c30f-4307-b414-57b4ee6bd313", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139672025-04-30T06:36:06.908ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
139682025-04-30T06:36:06.910ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "e8cd4e12-1796-4d47-aea2-4b916ec4bea6", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139692025-04-30T06:36:06.910ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
139702025-04-30T06:36:06.911ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "94c6987c-1718-469e-8ee3-4dfa00144798", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139712025-04-30T06:36:06.912ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
139722025-04-30T06:36:06.947ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
139732025-04-30T06:36:06.947ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
139742025-04-30T06:36:06.947ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139752025-04-30T06:36:06.948ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
139762025-04-30T06:36:06.949ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139772025-04-30T06:36:06.958ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
139782025-04-30T06:36:06.970ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
139792025-04-30T06:36:06.970ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139802025-04-30T06:36:06.970ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
139812025-04-30T06:36:06.989ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139822025-04-30T06:36:06.998ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
139832025-04-30T06:36:06.998ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
139842025-04-30T06:36:07.000ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
139852025-04-30T06:36:07.000ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b4a64d2-e928-4e2e-a029-f92b6f339b95", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139862025-04-30T06:36:07.000ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
139872025-04-30T06:36:07.011ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "51786a22-7e1e-4303-ba79-6aa7276c63d1", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139882025-04-30T06:36:07.011ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
139892025-04-30T06:36:07.025ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33ab7a36-4fff-4940-942b-fce798a465fa", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139902025-04-30T06:36:07.025ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
139912025-04-30T06:36:07.029ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "32a189b7-bafc-467b-8c7c-4841b21ca5be", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139922025-04-30T06:36:07.029ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
139932025-04-30T06:36:07.046ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b587e939-a385-463f-bef9-01dd6bf867bd", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139942025-04-30T06:36:07.046ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
139952025-04-30T06:36:07.050ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f133a4af-992c-4ef4-969b-ebd99743dd6a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139962025-04-30T06:36:07.050ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
139972025-04-30T06:36:07.051ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "37c612a1-21ff-4230-8b22-68a9ae68c3b4", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
139982025-04-30T06:36:07.051ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
139992025-04-30T06:36:07.067ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "235a1a39-e01c-4d5b-82db-4490cba997d1", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140002025-04-30T06:36:07.067ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
140012025-04-30T06:36:07.069ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
140022025-04-30T06:36:07.075ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "71f90bfb-ccb6-4552-a72c-9fc5ed7d7360", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140032025-04-30T06:36:07.076ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
140042025-04-30T06:36:07.083ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "2eacdabe-ddce-499a-af59-4def0bd498fd", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140052025-04-30T06:36:07.083ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
140062025-04-30T06:36:07.084ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4d014c2f-4cae-4449-af72-f581772915f3", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140072025-04-30T06:36:07.085ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
140082025-04-30T06:36:07.087ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b93c4cee-7dab-4ee2-8138-633300bea937", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140092025-04-30T06:36:07.087ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
140102025-04-30T06:36:07.089ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "721c94c3-b5c7-4662-bd9e-4244d1f77361", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140112025-04-30T06:36:07.089ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
140122025-04-30T06:36:07.093ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "a022a99f-8016-4787-aac6-ef63d3d10b95", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140132025-04-30T06:36:07.093ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
140142025-04-30T06:36:07.114ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9f391a72-b523-4fad-aa7a-a9ee55f28441", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140152025-04-30T06:36:07.114ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
140162025-04-30T06:36:07.116ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "d5197c19-c9ee-4797-a6fc-81bb2ea6f5f7", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140172025-04-30T06:36:07.116ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
140182025-04-30T06:36:07.119ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140192025-04-30T06:36:07.119ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140202025-04-30T06:36:07.119ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140212025-04-30T06:36:07.119ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140222025-04-30T06:36:07.120ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140232025-04-30T06:36:07.122ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c70f78e6-afaf-417b-a670-cd0ffa063086", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140242025-04-30T06:36:07.123ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
140252025-04-30T06:36:07.124ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ee4f02d-8054-4b4e-8998-d7c8713479ce", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140262025-04-30T06:36:07.125ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
140272025-04-30T06:36:07.126ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140282025-04-30T06:36:07.132ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140292025-04-30T06:36:07.133ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140302025-04-30T06:36:07.133ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140312025-04-30T06:36:07.134ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140322025-04-30T06:36:07.136ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6364c45-050e-40f7-9697-304eb0486991", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140332025-04-30T06:36:07.139ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
140342025-04-30T06:36:07.139ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140352025-04-30T06:36:07.139ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140362025-04-30T06:36:07.140ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140372025-04-30T06:36:07.142ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ae014329-4ed7-4b16-ab35-0e9dcccc0fda", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140382025-04-30T06:36:07.143ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
140392025-04-30T06:36:07.145ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "ce49cc7a-7fa2-406f-b2ef-ac8684a74798", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140402025-04-30T06:36:07.145ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
140412025-04-30T06:36:07.149ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "42206db1-8492-4437-9704-00052fdd4ccd", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140422025-04-30T06:36:07.149ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
140432025-04-30T06:36:07.154ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "421df839-47c2-446c-8af6-04a9c2f4e81a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140442025-04-30T06:36:07.154ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
140452025-04-30T06:36:07.158ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "de1d5208-86a6-4e02-9735-e02971edfe3a", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140462025-04-30T06:36:07.159ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
140472025-04-30T06:36:07.163ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "9c6c5922-524a-43e1-be08-2a8aee119cee", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140482025-04-30T06:36:07.164ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
140492025-04-30T06:36:07.167ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "4ff77266-ccce-4956-a4b0-722b44b3b912", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140502025-04-30T06:36:07.168ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
140512025-04-30T06:36:07.170ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ff45074c-743f-49ef-a126-38f49ab00083", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140522025-04-30T06:36:07.170ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
140532025-04-30T06:36:07.178ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
140542025-04-30T06:36:07.179ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "cbd1300c-3660-4e47-a19f-516df07916fc", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140552025-04-30T06:36:07.179ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
140562025-04-30T06:36:07.181ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "838fde11-cf45-4bc1-8865-4ce769510fb6", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140572025-04-30T06:36:07.186ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
140582025-04-30T06:36:07.189ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4a427f9d-b519-4be0-a47f-a64debae2259", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140592025-04-30T06:36:07.190ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
140602025-04-30T06:36:07.194ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b44486c9-4c88-4011-92da-837a9cf46bf7", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140612025-04-30T06:36:07.194ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
140622025-04-30T06:36:07.195ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "48c4bb7d-a505-458c-9b0c-7524fcc8b1de", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
140632025-04-30T06:36:07.195ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140642025-04-30T06:36:07.195ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140652025-04-30T06:36:07.195ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140662025-04-30T06:36:07.195ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140672025-04-30T06:36:07.196ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140682025-04-30T06:36:07.200ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140692025-04-30T06:36:07.207ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140702025-04-30T06:36:07.207ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
140712025-04-30T06:36:07.207ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140722025-04-30T06:36:07.207ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140732025-04-30T06:36:07.207ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140742025-04-30T06:36:07.208ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140752025-04-30T06:36:07.214ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140762025-04-30T06:36:07.228ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
140772025-04-30T06:36:07.228ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
140782025-04-30T06:36:07.228ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
140792025-04-30T06:36:07.228ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
140802025-04-30T06:36:07.229ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
140812025-04-30T06:36:07.229ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
140822025-04-30T06:36:07.229ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140832025-04-30T06:36:07.229ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140842025-04-30T06:36:07.229ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140852025-04-30T06:36:07.233ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
140862025-04-30T06:36:07.239ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
140872025-04-30T06:36:07.239ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140882025-04-30T06:36:07.239ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140892025-04-30T06:36:07.240ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140902025-04-30T06:36:07.310ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
140912025-04-30T06:36:07.310ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
140922025-04-30T06:36:07.311ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:38689 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
140932025-04-30T06:36:07.311ZINFOtest_metrics (oximeter-agent): refreshing list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:562
140942025-04-30T06:36:07.312ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
140952025-04-30T06:36:07.312ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
140962025-04-30T06:36:07.312ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
140972025-04-30T06:36:07.313ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
140982025-04-30T06:36:07.314ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 9e3c7f23-ded2-4842-907d-9126abf83ae0 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
140992025-04-30T06:36:07.314ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:38689 req_id = 3150c1d1-c2ad-4680-99bf-ec2ac4017abd sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141002025-04-30T06:36:07.315ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 9e3c7f23-ded2-4842-907d-9126abf83ae0 resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
141012025-04-30T06:36:07.315ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 9e3c7f23-ded2-4842-907d-9126abf83ae0 roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
141022025-04-30T06:36:07.315ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 9e3c7f23-ded2-4842-907d-9126abf83ae0 resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
141032025-04-30T06:36:07.316ZINFOtest_metrics (dropshot): request completed latency_us = 2243 local_addr = [::1]:56237 method = GET remote_addr = [::1]:38689 req_id = 3150c1d1-c2ad-4680-99bf-ec2ac4017abd response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
141042025-04-30T06:36:07.316ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "3150c1d1-c2ad-4680-99bf-ec2ac4017abd", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141052025-04-30T06:36:07.320ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141062025-04-30T06:36:07.320ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
141072025-04-30T06:36:07.320ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
141082025-04-30T06:36:07.322ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141092025-04-30T06:36:07.322ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141102025-04-30T06:36:07.323ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141112025-04-30T06:36:07.323ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141122025-04-30T06:36:07.323ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141132025-04-30T06:36:07.327ZINFOtest_metrics (dropshot_internal): request completed latency_us = 13883 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 9e3c7f23-ded2-4842-907d-9126abf83ae0 response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending
141142025-04-30T06:36:07.334ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?sort_by=id_ascending", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e3c7f23-ded2-4842-907d-9126abf83ae0", "content-length": "405", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141152025-04-30T06:36:07.335ZDEBGtest_metrics (oximeter): client request body = None method = GET uri = http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
141162025-04-30T06:36:07.335ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141172025-04-30T06:36:07.341ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141182025-04-30T06:36:07.341ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141192025-04-30T06:36:07.341ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141202025-04-30T06:36:07.343ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141212025-04-30T06:36:07.344ZTRACtest_metrics (dropshot_internal): incoming request local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 363172dd-7347-4f5d-b993-eed306d37c2e uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
141222025-04-30T06:36:07.346ZTRACtest_metrics (dropshot_internal): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 363172dd-7347-4f5d-b993-eed306d37c2e resource = Database uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
141232025-04-30T06:36:07.346ZDEBGtest_metrics (dropshot_internal): roles actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 363172dd-7347-4f5d-b993-eed306d37c2e roles = RoleSet { roles: {} } uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
141242025-04-30T06:36:07.347ZDEBGtest_metrics (dropshot_internal): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) actor_id = 001de000-05e4-4000-8000-000000000002 authenticated = true local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 363172dd-7347-4f5d-b993-eed306d37c2e resource = Database result = Ok(()) uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
141252025-04-30T06:36:07.350ZINFOtest_metrics (dropshot_internal): request completed latency_us = 5973 local_addr = 127.0.0.1:49095 method = GET remote_addr = 127.0.0.1:36706 req_id = 363172dd-7347-4f5d-b993-eed306d37c2e response_code = 200 uri = /metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D
141262025-04-30T06:36:07.357ZDEBGtest_metrics (oximeter): client response result = Ok(Response { url: "http://127.0.0.1:49095/metrics/collectors/39e6175b-4df2-4730-b11d-cbc1e60a2e78/producers?page_token=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJpZF9hc2NlbmRpbmciLCJsYXN0X3NlZW4iOiJhNjQ1OGI3ZC04N2MzLTQ0ODMtYmU5Ni04NTRkODE0YzIwZGUifX0%3D", status: 200, headers: {"content-type": "application/json", "x-request-id": "363172dd-7347-4f5d-b993-eed306d37c2e", "content-length": "29", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141272025-04-30T06:36:07.357ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = 588db90d-5640-4be3-9102-2ebf0af0c9a3
141282025-04-30T06:36:07.357ZTRACtest_metrics (oximeter-agent): ignoring request to update existing metric producer, since the endpoint information is the same as the existing collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141292025-04-30T06:36:07.357ZINFOtest_metrics (oximeter-agent): refreshed list of producers from Nexus collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 file = oximeter/collector/src/agent.rs:637 n_current_tasks = 2 n_pruned_tasks = 0
141302025-04-30T06:36:07.441ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
141312025-04-30T06:36:07.441ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
141322025-04-30T06:36:07.442ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
141332025-04-30T06:36:07.442ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 1.480378127s iteration = 50 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
141342025-04-30T06:36:07.442ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 51 reason = Signaled
141352025-04-30T06:36:07.442ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
141362025-04-30T06:36:07.442ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141372025-04-30T06:36:07.442ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141382025-04-30T06:36:07.442ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141392025-04-30T06:36:07.443ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141402025-04-30T06:36:07.447ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141412025-04-30T06:36:07.451ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
141422025-04-30T06:36:07.452ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
141432025-04-30T06:36:07.452ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
141442025-04-30T06:36:07.452ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
141452025-04-30T06:36:07.458ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
141462025-04-30T06:36:07.458ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
141472025-04-30T06:36:07.460ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: 95 f9 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:52873 req_id = 799f8107-ddbf-4398-a3f0-4bcde2782a92
141482025-04-30T06:36:07.460ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 38393,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:52873 req_id = 799f8107-ddbf-4398-a3f0-4bcde2782a92
141492025-04-30T06:36:07.460ZDEBGtest_metrics (store): query key key = _mgs._tcp
141502025-04-30T06:36:07.460ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141512025-04-30T06:36:07.461ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:52873 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 799f8107-ddbf-4398-a3f0-4bcde2782a92
141522025-04-30T06:36:07.461ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3832, tv_nsec: 260277834 } })
141532025-04-30T06:36:07.461ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
141542025-04-30T06:36:07.461ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: b5 4c 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e .L..........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:60425 req_id = 7d91aca0-28ea-489a-8efc-98bf582eed4c
141552025-04-30T06:36:07.461ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 46412,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:60425 req_id = 7d91aca0-28ea-489a-8efc-98bf582eed4c
141562025-04-30T06:36:07.461ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
141572025-04-30T06:36:07.461ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:60425 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 7d91aca0-28ea-489a-8efc-98bf582eed4c
141582025-04-30T06:36:07.524ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
141592025-04-30T06:36:07.525ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
141602025-04-30T06:36:07.525ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
141612025-04-30T06:36:07.525ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141622025-04-30T06:36:07.525ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
141632025-04-30T06:36:07.526ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141642025-04-30T06:36:07.526ZDEBGtest_metrics (collection-task): sent timer-based collection request to the collection task collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141652025-04-30T06:36:07.526ZDEBGtest_metrics (collection-task): collecting from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141662025-04-30T06:36:07.527ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 15 fe 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c ............._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:53712 req_id = 65a82e66-5138-4307-bfe1-bd5e2eea1286
141672025-04-30T06:36:07.527ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 5630,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:53712 req_id = 65a82e66-5138-4307-bfe1-bd5e2eea1286
141682025-04-30T06:36:07.527ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
141692025-04-30T06:36:07.527ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 5630, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:53712 req_id = 65a82e66-5138-4307-bfe1-bd5e2eea1286
141702025-04-30T06:36:07.529ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
141712025-04-30T06:36:07.529ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
141722025-04-30T06:36:07.529ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
141732025-04-30T06:36:07.530ZTRACtest_metrics (collection-task): sent collection request to producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141742025-04-30T06:36:07.531ZDEBGtest_metrics (collection-task): collected results from producer collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 n_results = 1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141752025-04-30T06:36:07.531ZTRACtest_metrics (collection-task): forwarded results to main collection loop collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141762025-04-30T06:36:07.531ZTRACtest_metrics (collection-task): top of inner collection loop, waiting for next request collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 producer_id = a6458b7d-87c3-4483-be96-854d814c20de
141772025-04-30T06:36:07.531ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
141782025-04-30T06:36:07.537ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
141792025-04-30T06:36:07.537ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141802025-04-30T06:36:07.537ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
141812025-04-30T06:36:07.538ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141822025-04-30T06:36:07.540ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "9e8f0560-e150-4d82-824a-489f71d26079", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141832025-04-30T06:36:07.542ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
141842025-04-30T06:36:07.542ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
141852025-04-30T06:36:07.542ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
141862025-04-30T06:36:07.542ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
141872025-04-30T06:36:07.546ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9f54008-9fd6-4e39-b180-1f9fba8297a6", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141882025-04-30T06:36:07.546ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
141892025-04-30T06:36:07.554ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2c855bc6-a92e-4d66-ac58-aff0fac80c1a", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141902025-04-30T06:36:07.554ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
141912025-04-30T06:36:07.558ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b9a068cd-0627-4e5b-912e-013c52988de9", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141922025-04-30T06:36:07.558ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
141932025-04-30T06:36:07.560ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "595ae3f4-5355-4c57-9949-55d33b5c37d7", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141942025-04-30T06:36:07.561ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
141952025-04-30T06:36:07.564ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "b6433ba9-01a3-4287-a3db-139975377ef3", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141962025-04-30T06:36:07.564ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
141972025-04-30T06:36:07.567ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "ee615332-bc64-419a-b8a8-cc652b887945", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
141982025-04-30T06:36:07.567ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
141992025-04-30T06:36:07.567ZDEBGtest_metrics (results-sink): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
142002025-04-30T06:36:07.567ZDEBGtest_metrics (results-sink-cluster): inserting 2 samples into database collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
142012025-04-30T06:36:07.568ZDEBGtest_metrics (clickhouse-client): unrolling 2 total samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e
142022025-04-30T06:36:07.569ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.fields_string FORMAT Native
142032025-04-30T06:36:07.574ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = fields_string
142042025-04-30T06:36:07.574ZTRACtest_metrics (clickhouse-client): inserting data collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_columns = 4 n_rows = 2 sql = INSERT INTO oximeter.measurements_i64 FORMAT Native
142052025-04-30T06:36:07.575ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "decfa7c0-9598-44b7-980a-1c52718fa93a", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142062025-04-30T06:36:07.575ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
142072025-04-30T06:36:07.576ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
142082025-04-30T06:36:07.579ZDEBGtest_metrics (clickhouse-client): inserted rows into table collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 id = 758cdc67-47ef-43f0-b1af-382db611bb3e n_rows = 2 table_name = measurements_i64
142092025-04-30T06:36:07.579ZTRACtest_metrics (results-sink): successfully inserted samples collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1
142102025-04-30T06:36:07.579ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "cda0607d-f532-48c1-bf93-425cb0ed77f5", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142112025-04-30T06:36:07.580ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
142122025-04-30T06:36:07.581ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "fb977060-7760-45ee-99f5-a0273bec66cc", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142132025-04-30T06:36:07.582ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
142142025-04-30T06:36:07.584ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d6fc024a-5fdd-4bc1-ac4f-e321ec27196a", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142152025-04-30T06:36:07.584ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
142162025-04-30T06:36:07.586ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "525f6311-297f-434e-a57c-b82f581f4ee5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142172025-04-30T06:36:07.587ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
142182025-04-30T06:36:07.591ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "f07cd998-7042-4654-929a-bc593e75ad53", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142192025-04-30T06:36:07.591ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
142202025-04-30T06:36:07.594ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f3eed910-39e0-4adc-843b-4496bd6bf6b8", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142212025-04-30T06:36:07.594ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
142222025-04-30T06:36:07.596ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18add4de-04d3-4bf3-b3b8-2ed2d5d9cc39", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142232025-04-30T06:36:07.596ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0
142242025-04-30T06:36:07.599ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33a58f55-47d8-46e1-9745-c12f2f844a56", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142252025-04-30T06:36:07.599ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1
142262025-04-30T06:36:07.601ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "2323dde3-0f4b-47e8-b5e2-7f00f893a864", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142272025-04-30T06:36:07.601ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0
142282025-04-30T06:36:07.603ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f715b563-572b-4b06-9c94-431872adc132", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142292025-04-30T06:36:07.604ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1
142302025-04-30T06:36:07.607ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "eebc4898-c86c-4684-99d5-5ef363101c2d", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142312025-04-30T06:36:07.607ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cmpa
142322025-04-30T06:36:07.610ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "878d136e-eb24-4b3a-82b9-ef519c50609d", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142332025-04-30T06:36:07.610ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
142342025-04-30T06:36:07.611ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "9b1df641-184d-45da-8564-eff4c458e501", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142352025-04-30T06:36:07.611ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
142362025-04-30T06:36:07.612ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f2e6f315-9639-42f8-b1b5-770e898a9926", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142372025-04-30T06:36:07.613ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/cfpa
142382025-04-30T06:36:07.614ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "3f2214d9-0a04-40b4-943e-10ed7fffa716", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142392025-04-30T06:36:07.614ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0
142402025-04-30T06:36:07.615ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "bf931819-7816-4067-a557-73147cfcdce5", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142412025-04-30T06:36:07.615ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0
142422025-04-30T06:36:07.617ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "33af51f7-b40e-470a-9f1a-577be05236a0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142432025-04-30T06:36:07.617ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1
142442025-04-30T06:36:07.620ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "12f5fb82-a5a4-480a-8580-e8d68e5998b0", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142452025-04-30T06:36:07.620ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0
142462025-04-30T06:36:07.622ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "19d4a15e-768c-4817-9c1b-324e5c1a512d", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142472025-04-30T06:36:07.622ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1
142482025-04-30T06:36:07.625ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "7278429b-9cf3-4da0-9c3a-a298dcc6e2b5", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142492025-04-30T06:36:07.625ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0
142502025-04-30T06:36:07.639ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
142512025-04-30T06:36:07.639ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
142522025-04-30T06:36:07.639ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142532025-04-30T06:36:07.639ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
142542025-04-30T06:36:07.644ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142552025-04-30T06:36:07.646ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "4f8efc80-48c2-4783-9cd6-d14e2e5cefc8", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:06 GMT"} })
142562025-04-30T06:36:07.646ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1
142572025-04-30T06:36:07.647ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
142582025-04-30T06:36:07.659ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
142592025-04-30T06:36:07.660ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142602025-04-30T06:36:07.660ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
142612025-04-30T06:36:07.660ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142622025-04-30T06:36:07.663ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "00b5527b-e326-462c-89c2-39014ca1b2f2", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142632025-04-30T06:36:07.663ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cmpa
142642025-04-30T06:36:07.663ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
142652025-04-30T06:36:07.663ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
142662025-04-30T06:36:07.664ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
142672025-04-30T06:36:07.675ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a793c64c-6c6a-4639-b93e-a188328ffcb9", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142682025-04-30T06:36:07.676ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
142692025-04-30T06:36:07.682ZWARNtest_metrics (results-sink-cluster): failed to insert some results into metric DB collector_id = 39e6175b-4df2-4730-b11d-cbc1e60a2e78 collector_ip = ::1 error = Failed to check out connection to database: No backends found for this service file = oximeter/collector/src/results_sink.rs:92
142702025-04-30T06:36:07.684ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "c0d90d2a-6e57-4080-ba5c-c15900faf7ae", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142712025-04-30T06:36:07.684ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
142722025-04-30T06:36:07.687ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "84647ce2-d9fd-494b-9c2f-cc24d2ef6500", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142732025-04-30T06:36:07.687ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/0/component/rot/cfpa
142742025-04-30T06:36:07.688ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b8443272-a31b-441c-8536-94a8c8ac78ee", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142752025-04-30T06:36:07.688ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1
142762025-04-30T06:36:07.691ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "653d1974-61f8-4e64-9179-5714c5172c92", "content-length": "723", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142772025-04-30T06:36:07.691ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0
142782025-04-30T06:36:07.699ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "2fa3a35d-6096-4a14-a918-913f3bf29a71", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142792025-04-30T06:36:07.699ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1
142802025-04-30T06:36:07.703ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "18c11faa-4bb2-4c2f-9654-966628b1573d", "content-length": "109", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142812025-04-30T06:36:07.703ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0
142822025-04-30T06:36:07.706ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "f019d6a7-4269-4aa4-b682-593692088320", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142832025-04-30T06:36:07.706ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1
142842025-04-30T06:36:07.711ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "c6d45369-6963-47db-8aea-81eb1f904006", "content-length": "169", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142852025-04-30T06:36:07.711ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0
142862025-04-30T06:36:07.716ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
142872025-04-30T06:36:07.719ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "1368412d-3122-4767-8d6b-89508bba7af1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142882025-04-30T06:36:07.720ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1
142892025-04-30T06:36:07.724ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "9867cfb1-f502-46ec-8f28-1ff0e67427d1", "content-length": "178", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142902025-04-30T06:36:07.724ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cmpa
142912025-04-30T06:36:07.725ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "4786cab7-8174-4898-94c7-083fb9c1d254", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142922025-04-30T06:36:07.726ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
142932025-04-30T06:36:07.728ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "80753c9d-589e-40b2-b465-bf1b0a01cee0", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142942025-04-30T06:36:07.728ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
142952025-04-30T06:36:07.729ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "f9a25ebe-de36-4bce-a289-831f3351ec4d", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142962025-04-30T06:36:07.729ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/sled/1/component/rot/cfpa
142972025-04-30T06:36:07.733ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/sled/1/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "0468c54b-69c9-47c2-9624-4b58b3756eb5", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
142982025-04-30T06:36:07.733ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
142992025-04-30T06:36:07.733ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143002025-04-30T06:36:07.733ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143012025-04-30T06:36:07.733ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143022025-04-30T06:36:07.734ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143032025-04-30T06:36:07.739ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143042025-04-30T06:36:07.745ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143052025-04-30T06:36:07.745ZTRACtest_metrics (BackgroundTasks): authorize begin action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet
143062025-04-30T06:36:07.745ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143072025-04-30T06:36:07.745ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143082025-04-30T06:36:07.745ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143092025-04-30T06:36:07.746ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143102025-04-30T06:36:07.748ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143112025-04-30T06:36:07.753ZDEBGtest_metrics (BackgroundTasks): authorize result action = ListChildren actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Fleet result = Ok(())
143122025-04-30T06:36:07.753ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143132025-04-30T06:36:07.753ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143142025-04-30T06:36:07.754ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143152025-04-30T06:36:07.833ZDEBGtest_metrics (BackgroundTasks): begin collection from Sled Agent background_task = inventory_collection sled_agent_url = http://[::1]:56237
143162025-04-30T06:36:07.833ZDEBGtest_metrics (BackgroundTasks): client request SledAgent = http://[::1]:56237 background_task = inventory_collection body = None method = GET uri = http://[::1]:56237/inventory
143172025-04-30T06:36:07.834ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143182025-04-30T06:36:07.834ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143192025-04-30T06:36:07.835ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143202025-04-30T06:36:07.835ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143212025-04-30T06:36:07.835ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143222025-04-30T06:36:07.836ZINFOtest_metrics (dropshot): accepted connection local_addr = [::1]:56237 remote_addr = [::1]:60421 sled_id = b6d65341-167c-41df-9b5c-41cded99c229
143232025-04-30T06:36:07.836ZTRACtest_metrics (dropshot): incoming request local_addr = [::1]:56237 method = GET remote_addr = [::1]:60421 req_id = 9ac198bf-d7c3-48e6-a27d-55aaa5f13de6 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143242025-04-30T06:36:07.838ZINFOtest_metrics (dropshot): request completed latency_us = 1431 local_addr = [::1]:56237 method = GET remote_addr = [::1]:60421 req_id = 9ac198bf-d7c3-48e6-a27d-55aaa5f13de6 response_code = 200 sled_id = b6d65341-167c-41df-9b5c-41cded99c229 uri = /inventory
143252025-04-30T06:36:07.838ZDEBGtest_metrics (BackgroundTasks): client response SledAgent = http://[::1]:56237 background_task = inventory_collection result = Ok(Response { url: "http://[::1]:56237/inventory", status: 200, headers: {"content-type": "application/json", "x-request-id": "9ac198bf-d7c3-48e6-a27d-55aaa5f13de6", "content-length": "3141", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
143262025-04-30T06:36:07.846ZDEBGtest_metrics (BackgroundTasks): begin collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143272025-04-30T06:36:07.846ZDEBGtest_metrics (BackgroundTasks): end collecting all keepers background_task = inventory_collection nkeeper_admin_clients = 0
143282025-04-30T06:36:07.846ZDEBGtest_metrics (BackgroundTasks): finished collection background_task = inventory_collection
143292025-04-30T06:36:07.849ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143302025-04-30T06:36:07.849ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143312025-04-30T06:36:07.849ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143322025-04-30T06:36:07.849ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143332025-04-30T06:36:07.850ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143342025-04-30T06:36:07.851ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143352025-04-30T06:36:07.858ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143362025-04-30T06:36:07.859ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143372025-04-30T06:36:07.859ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143382025-04-30T06:36:07.859ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143392025-04-30T06:36:07.862ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143402025-04-30T06:36:07.862ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143412025-04-30T06:36:07.863ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143422025-04-30T06:36:07.870ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143432025-04-30T06:36:07.875ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143442025-04-30T06:36:07.875ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143452025-04-30T06:36:07.875ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143462025-04-30T06:36:07.876ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143472025-04-30T06:36:07.954ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
143482025-04-30T06:36:07.954ZWARNtest_metrics (datastore): Non-retryable transaction failure err = DatabaseError(UniqueViolation, "duplicate key value violates unique constraint \\"inv_caboose_pkey\\"") transaction = inventory_insert_collection
143492025-04-30T06:36:07.955ZWARNtest_metrics (BackgroundTasks): inventory collection failed background_task = inventory_collection error = failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint "inv_caboose_pkey"
143502025-04-30T06:36:07.955ZDEBGtest_metrics (BackgroundTasks): activation complete background_task = inventory_collection elapsed = 512.720852ms iteration = 51 status = {"error":"failed to collect inventory: saving inventory to database: Internal Error: unexpected database error: duplicate key value violates unique constraint \\"inv_caboose_pkey\\""}
143512025-04-30T06:36:07.955ZDEBGtest_metrics (BackgroundTasks): activating background_task = inventory_collection iteration = 52 reason = Signaled
143522025-04-30T06:36:07.955ZTRACtest_metrics (BackgroundTasks): authorize begin action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory
143532025-04-30T06:36:07.955ZTRACtest_metrics (BackgroundTasks): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } background_task = inventory_collection resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143542025-04-30T06:36:07.955ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143552025-04-30T06:36:07.955ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143562025-04-30T06:36:07.956ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143572025-04-30T06:36:07.967ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143582025-04-30T06:36:07.971ZDEBGtest_metrics (BackgroundTasks): authorize result action = Modify actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Inventory result = Ok(())
143592025-04-30T06:36:07.974ZTRACtest_metrics (BackgroundTasks): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database
143602025-04-30T06:36:07.974ZDEBGtest_metrics (BackgroundTasks): roles background_task = inventory_collection roles = RoleSet { roles: {} }
143612025-04-30T06:36:07.975ZDEBGtest_metrics (BackgroundTasks): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) background_task = inventory_collection resource = Database result = Ok(())
143622025-04-30T06:36:07.985ZDEBGtest_metrics (BackgroundTasks): inventory_prune_one: nothing eligible for removal (too few) background_task = inventory_collection candidates = [(2faaadaa-5e50-408d-a18e-243c0932f65f, 0)]
143632025-04-30T06:36:07.985ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal
143642025-04-30T06:36:07.987ZTRACtest_metrics (dns): buffer buffer = Length: 67 (0x43) bytes\n0000: ca 09 01 00 00 01 00 00 00 00 00 01 04 5f 6d 67 ............._mg\n0010: 73 04 5f 74 63 70 0d 63 6f 6e 74 72 6f 6c 2d 70 s._tcp.control-p\n0020: 6c 61 6e 65 05 6f 78 69 64 65 08 69 6e 74 65 72 lane.oxide.inter\n0030: 6e 61 6c 00 00 21 00 01 00 00 29 04 d0 00 00 00 nal..!....).....\n0040: 00 00 00 ... kind = dns peer_addr = [::1]:55047 req_id = a5f9b967-5af7-4628-ae6d-38fef7170f27
143652025-04-30T06:36:07.987ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 51721,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_mgs._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_mgs._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 4,\n 95,\n 109,\n 103,\n 115,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55047 req_id = a5f9b967-5af7-4628-ae6d-38fef7170f27
143662025-04-30T06:36:07.988ZDEBGtest_metrics (store): query key key = _mgs._tcp
143672025-04-30T06:36:07.991ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143682025-04-30T06:36:07.991ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:55047 query = LowerQuery { name: LowerName(Name("_mgs._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_mgs._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = a5f9b967-5af7-4628-ae6d-38fef7170f27
143692025-04-30T06:36:07.992ZDEBGtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _mgs._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_mgs._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_mgs._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 36378, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3832, tv_nsec: 791050463 } })
143702025-04-30T06:36:07.992ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
143712025-04-30T06:36:07.992ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 86 d6 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e ............-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:58769 req_id = 56bf3176-dcad-47e0-a15c-af667840268e
143722025-04-30T06:36:07.992ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 34518,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:58769 req_id = 56bf3176-dcad-47e0-a15c-af667840268e
143732025-04-30T06:36:07.992ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
143742025-04-30T06:36:07.994ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:58769 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = 56bf3176-dcad-47e0-a15c-af667840268e
143752025-04-30T06:36:08.101ZTRACtest_metrics (DnsResolver): lookup_all_socket_v6 srv dns_name = _clickhouse-admin-keeper._tcp.control-plane.oxide.internal
143762025-04-30T06:36:08.101ZTRACtest_metrics (InternalApi): authorize begin action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory
143772025-04-30T06:36:08.101ZTRACtest_metrics (InternalApi): loading roles actor = Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. } resource_id = 001de000-1334-4000-8000-000000000000 resource_type = Fleet
143782025-04-30T06:36:08.101ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143792025-04-30T06:36:08.101ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143802025-04-30T06:36:08.102ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143812025-04-30T06:36:08.103ZTRACtest_metrics (dns): buffer buffer = Length: 87 (0x57) bytes\n0000: 45 2c 01 00 00 01 00 00 00 00 00 01 18 5f 63 6c E,..........._cl\n0010: 69 63 6b 68 6f 75 73 65 2d 61 64 6d 69 6e 2d 6b ickhouse-admin-k\n0020: 65 65 70 65 72 04 5f 74 63 70 0d 63 6f 6e 74 72 eeper._tcp.contr\n0030: 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 69 ol-plane.oxide.i\n0040: 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 04 nternal..!....).\n0050: d0 00 00 00 00 00 00 ....... kind = dns peer_addr = [::1]:56379 req_id = 93132843-5cbd-4f96-a967-89b3dc9cf9ab
143822025-04-30T06:36:08.103ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 17708,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 24,\n 95,\n 99,\n 108,\n 105,\n 99,\n 107,\n 104,\n 111,\n 117,\n 115,\n 101,\n 45,\n 97,\n 100,\n 109,\n 105,\n 110,\n 45,\n 107,\n 101,\n 101,\n 112,\n 101,\n 114,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:56379 req_id = 93132843-5cbd-4f96-a967-89b3dc9cf9ab
143832025-04-30T06:36:08.103ZDEBGtest_metrics (store): query key key = _clickhouse-admin-keeper._tcp
143842025-04-30T06:36:08.103ZERROtest_metrics (dns): failed to handle incoming DNS message: MessageRequest { header: Header { id: 17708, message_type: Query, op_code: Query, authoritative: false, truncation: false, recursion_desired: true, recursion_available: false, authentic_data: false, checking_disabled: false, response_code: NoError, query_count: 1, answer_count: 0, name_server_count: 0, additional_count: 1, }, query: WireQuery { query: LowerQuery { name: LowerName( Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), ), original: Query { name: Name("_clickhouse-admin-keeper._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN, }, }, original: [ 24, 95, 99, 108, 105, 99, 107, 104, 111, 117, 115, 101, 45, 97, 100, 109, 105, 110, 45, 107, 101, 101, 112, 101, 114, 4, 95, 116, 99, 112, 13, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 5, 111, 120, 105, 100, 101, 8, 105, 110, 116, 101, 114, 110, 97, 108, 0, 0, 33, 0, 1, ], }, answers: [], name_servers: [], additionals: [], sig0: [], edns: Some( Edns { rcode_high: 0, version: 0, dnssec_ok: false, max_payload: 1232, options: OPT { options: {}, }, }, ), } NXDOMAIN: no records found for name: "_clickhouse-admin-keeper._tcp.control-plane.oxide.internal." kind = dns peer_addr = [::1]:56379 req_id = 93132843-5cbd-4f96-a967-89b3dc9cf9ab
143852025-04-30T06:36:08.104ZDEBGtest_metrics (BackgroundTasks): begin collection background_task = inventory_collection
143862025-04-30T06:36:08.104ZDEBGtest_metrics (BackgroundTasks): begin collection from MGS background_task = inventory_collection mgs_url = http://[::1]:36378
143872025-04-30T06:36:08.104ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/ignition
143882025-04-30T06:36:08.106ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {(Fleet, 001de000-1334-4000-8000-000000000000, "admin")} }
143892025-04-30T06:36:08.114ZDEBGtest_metrics (InternalApi): authorize result action = Read actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Inventory result = Ok(())
143902025-04-30T06:36:08.114ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143912025-04-30T06:36:08.114ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143922025-04-30T06:36:08.114ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143932025-04-30T06:36:08.116ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/ignition", status: 200, headers: {"content-type": "application/json", "x-request-id": "f37693c4-36a9-47f3-9b50-b2baf8270f52", "content-length": "835", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
143942025-04-30T06:36:08.116ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0
143952025-04-30T06:36:08.118ZTRACtest_metrics (InternalApi): authorize begin action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database
143962025-04-30T06:36:08.118ZDEBGtest_metrics (InternalApi): roles roles = RoleSet { roles: {} }
143972025-04-30T06:36:08.119ZDEBGtest_metrics (InternalApi): authorize result action = Query actor = Some(Actor::UserBuiltin { user_builtin_id: 001de000-05e4-4000-8000-000000000002, .. }) resource = Database result = Ok(())
143982025-04-30T06:36:08.123ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0", status: 200, headers: {"content-type": "application/json", "x-request-id": "5f8ef9d4-d773-4c44-a0ea-8bf0d3f7d135", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
143992025-04-30T06:36:08.124ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0
144002025-04-30T06:36:08.129ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "c9fdca5c-c77d-405b-9f03-079551c04cc2", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144012025-04-30T06:36:08.130ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1
144022025-04-30T06:36:08.136ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "56c2610f-731d-4760-984c-8abb6579b3cf", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144032025-04-30T06:36:08.136ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0
144042025-04-30T06:36:08.140ZTRACtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal
144052025-04-30T06:36:08.141ZTRACtest_metrics (dns): buffer buffer = Length: 72 (0x48) bytes\n0000: 3e 36 01 00 00 01 00 00 00 00 00 01 09 5f 64 65 >6..........._de\n0010: 6e 64 72 69 74 65 04 5f 74 63 70 0d 63 6f 6e 74 ndrite._tcp.cont\n0020: 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 69 64 65 08 rol-plane.oxide.\n0030: 69 6e 74 65 72 6e 61 6c 00 00 21 00 01 00 00 29 internal..!....)\n0040: 04 d0 00 00 00 00 00 00 ........ kind = dns peer_addr = [::1]:49739 req_id = 73163432-8d0f-411a-aefb-8ea6754b93b1
144062025-04-30T06:36:08.141ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 15926,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("_dendrite._tcp.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("_dendrite._tcp.control-plane.oxide.internal."),\n query_type: SRV,\n query_class: IN,\n },\n },\n original: [\n 9,\n 95,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 4,\n 95,\n 116,\n 99,\n 112,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 33,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:49739 req_id = 73163432-8d0f-411a-aefb-8ea6754b93b1
144072025-04-30T06:36:08.141ZDEBGtest_metrics (store): query key key = _dendrite._tcp
144082025-04-30T06:36:08.142ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144092025-04-30T06:36:08.142ZDEBGtest_metrics (dns): dns response additional_records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal"), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] kind = dns peer_addr = [::1]:49739 query = LowerQuery { name: LowerName(Name("_dendrite._tcp.control-plane.oxide.internal.")), original: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal."), query_type: SRV, query_class: IN } } records = [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal") })) }] req_id = 73163432-8d0f-411a-aefb-8ea6754b93b1
144102025-04-30T06:36:08.144ZDEBGtest_metrics (DnsResolver): lookup_all_ipv6 srv dns_name = _dendrite._tcp.control-plane.oxide.internal response = SrvLookup(Lookup { query: Query { name: Name("_dendrite._tcp.control-plane.oxide.internal"), query_type: SRV, query_class: IN }, records: [Record { name_labels: Name("_dendrite._tcp.control-plane.oxide.internal."), rr_type: SRV, dns_class: IN, ttl: 0, rdata: Some(SRV(SRV { priority: 0, weight: 0, port: 46964, target: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.") })) }, Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }], valid_until: Instant { tv_sec: 3832, tv_nsec: 942974381 } })
144112025-04-30T06:36:08.144ZTRACtest_metrics (DnsResolver): lookup_service_targets: looking up SRV target
144122025-04-30T06:36:08.145ZTRACtest_metrics (dns): buffer buffer = Length: 108 (0x6c) bytes\n0000: 79 e7 01 00 00 01 00 00 00 00 00 01 2d 64 65 6e y...........-den\n0010: 64 72 69 74 65 2d 62 36 64 36 35 33 34 31 2d 31 drite-b6d65341-1\n0020: 36 37 63 2d 34 31 64 66 2d 39 62 35 63 2d 34 31 67c-41df-9b5c-41\n0030: 63 64 65 64 39 39 63 32 32 39 04 68 6f 73 74 0d cded99c229.host.\n0040: 63 6f 6e 74 72 6f 6c 2d 70 6c 61 6e 65 05 6f 78 control-plane.ox\n0050: 69 64 65 08 69 6e 74 65 72 6e 61 6c 00 00 1c 00 ide.internal....\n0060: 01 00 00 29 04 d0 00 00 00 00 00 00 ...)........ kind = dns peer_addr = [::1]:55421 req_id = d046228b-550b-48fe-81c8-43830a33fa2c
144132025-04-30T06:36:08.145ZDEBGtest_metrics (dns): message_request kind = dns mr = MessageRequest {\n header: Header {\n id: 31207,\n message_type: Query,\n op_code: Query,\n authoritative: false,\n truncation: false,\n recursion_desired: true,\n recursion_available: false,\n authentic_data: false,\n checking_disabled: false,\n response_code: NoError,\n query_count: 1,\n answer_count: 0,\n name_server_count: 0,\n additional_count: 1,\n },\n query: WireQuery {\n query: LowerQuery {\n name: LowerName(\n Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n ),\n original: Query {\n name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."),\n query_type: AAAA,\n query_class: IN,\n },\n },\n original: [\n 45,\n 100,\n 101,\n 110,\n 100,\n 114,\n 105,\n 116,\n 101,\n 45,\n 98,\n 54,\n 100,\n 54,\n 53,\n 51,\n 52,\n 49,\n 45,\n 49,\n 54,\n 55,\n 99,\n 45,\n 52,\n 49,\n 100,\n 102,\n 45,\n 57,\n 98,\n 53,\n 99,\n 45,\n 52,\n 49,\n 99,\n 100,\n 101,\n 100,\n 57,\n 57,\n 99,\n 50,\n 50,\n 57,\n 4,\n 104,\n 111,\n 115,\n 116,\n 13,\n 99,\n 111,\n 110,\n 116,\n 114,\n 111,\n 108,\n 45,\n 112,\n 108,\n 97,\n 110,\n 101,\n 5,\n 111,\n 120,\n 105,\n 100,\n 101,\n 8,\n 105,\n 110,\n 116,\n 101,\n 114,\n 110,\n 97,\n 108,\n 0,\n 0,\n 28,\n 0,\n 1,\n ],\n },\n answers: [],\n name_servers: [],\n additionals: [],\n sig0: [],\n edns: Some(\n Edns {\n rcode_high: 0,\n version: 0,\n dnssec_ok: false,\n max_payload: 1232,\n options: OPT {\n options: {},\n },\n },\n ),\n} peer_addr = [::1]:55421 req_id = d046228b-550b-48fe-81c8-43830a33fa2c
144142025-04-30T06:36:08.145ZDEBGtest_metrics (store): query key key = dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host
144152025-04-30T06:36:08.145ZDEBGtest_metrics (dns): dns response additional_records = [] kind = dns peer_addr = [::1]:55421 query = LowerQuery { name: LowerName(Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal.")), original: Query { name: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), query_type: AAAA, query_class: IN } } records = [Record { name_labels: Name("dendrite-b6d65341-167c-41df-9b5c-41cded99c229.host.control-plane.oxide.internal."), rr_type: AAAA, dns_class: IN, ttl: 0, rdata: Some(AAAA(AAAA(::1))) }] req_id = d046228b-550b-48fe-81c8-43830a33fa2c
144162025-04-30T06:36:08.145ZINFOtest_metrics (BackgroundTasks): Determining switch slots managed by switch zones background_task = nat_v4_garbage_collector
144172025-04-30T06:36:08.189ZINFOtest_metrics (BackgroundTasks): determining switch slot managed by dendrite zone background_task = nat_v4_garbage_collector zone_address = ::1
144182025-04-30T06:36:08.189ZDEBGtest_metrics (MgsClient): client request background_task = nat_v4_garbage_collector body = None method = GET uri = http://[::1]:12225/local/switch-id
144192025-04-30T06:36:08.189ZDEBGtest_metrics (MgsClient): client response background_task = nat_v4_garbage_collector result = Err(reqwest::Error { kind: Request, url: "http://[::1]:12225/local/switch-id", source: hyper_util::client::legacy::Error(Connect, ConnectError("tcp connect error", Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) })
144202025-04-30T06:36:08.189ZWARNtest_metrics (BackgroundTasks): failed to identify switch slot for dendrite background_task = nat_v4_garbage_collector reason = Communication Error: error sending request for url (http://[::1]:12225/local/switch-id): client error (Connect): tcp connect error: Connection refused (os error 146): Connection refused (os error 146) zone_address = ::1
144212025-04-30T06:36:08.190ZWARNtest_metrics (BackgroundTasks): Failed to map switch zone addr: Communication Error: error sending request for url (http://[::1]:12225/local/switch-id), retrying background_task = nat_v4_garbage_collector
144222025-04-30T06:36:08.192ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "b3825ed0-1174-430f-87c4-aeefc1684ddc", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144232025-04-30T06:36:08.192ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1
144242025-04-30T06:36:08.200ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a73efb5a-1a9a-42ed-86d2-999efbfa1924", "content-length": "170", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144252025-04-30T06:36:08.202ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0
144262025-04-30T06:36:08.202ZDEBGtest_metrics: attempt to fetch latest inventory collection result = Ok("found")
144272025-04-30T06:36:08.207ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "36995c86-5adb-4fce-bb0f-51c2dd437ab1", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144282025-04-30T06:36:08.207ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1
144292025-04-30T06:36:08.211ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/stage0/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "a74056df-d9f2-4205-9787-b77dbbab079b", "content-length": "179", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144302025-04-30T06:36:08.211ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cmpa
144312025-04-30T06:36:08.213ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cmpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "05a1eedb-6ae5-4891-80ca-f0cca6c360d7", "content-length": "702", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144322025-04-30T06:36:08.213ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
144332025-04-30T06:36:08.215ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "a4e7a5bb-b415-4bf7-b639-4f0ea72339b6", "content-length": "727", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144342025-04-30T06:36:08.215ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
144352025-04-30T06:36:08.217ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "b53627e9-cf47-4573-b099-6b7f5768d0b0", "content-length": "729", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144362025-04-30T06:36:08.217ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = Some(Body) gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/0/component/rot/cfpa
144372025-04-30T06:36:08.219ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/0/component/rot/cfpa", status: 200, headers: {"content-type": "application/json", "x-request-id": "d973ecde-85f8-4825-b642-4f268f253963", "content-length": "728", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144382025-04-30T06:36:08.219ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1
144392025-04-30T06:36:08.221ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1", status: 200, headers: {"content-type": "application/json", "x-request-id": "899e7b26-1e01-4428-98ea-bd5aad07fb09", "content-length": "734", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144402025-04-30T06:36:08.221ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0
144412025-04-30T06:36:08.223ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=0", status: 200, headers: {"content-type": "application/json", "x-request-id": "20da5b79-f46f-4473-ab4b-140e29cfd2a0", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144422025-04-30T06:36:08.223ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1
144432025-04-30T06:36:08.254ZDEBGtest_metrics (BackgroundTasks): client response background_task = inventory_collection gateway_url = http://[::1]:36378 result = Ok(Response { url: "http://[::1]:36378/sp/switch/1/component/sp/caboose?firmware_slot=1", status: 200, headers: {"content-type": "application/json", "x-request-id": "63c7a7f6-0ff4-413e-8e7b-3689ad97ca07", "content-length": "111", "date": "Wed, 30 Apr 2025 06:36:07 GMT"} })
144442025-04-30T06:36:08.254ZDEBGtest_metrics (BackgroundTasks): client request background_task = inventory_collection body = None gateway_url = http://[::1]:36378 method = GET uri = http://[::1]:36378/sp/switch/1/component/rot/caboose?firmware_slot=0